{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_z2fw17x",
    "id": "64AE0FB342E9428A8DA9C734B357BBB1",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# softmax 和分类模型\n",
    "\n",
    "> 本文参考：\n",
    "- 伯禹AI：https://www.boyuai.com/elites/course/cZu18YmweLv10OeV\n",
    "- 动手学深度学习：http://zh.d2l.ai/\n",
    "- 动手学深度学习 TF 版：https://trickygo.github.io/Dive-into-DL-TensorFlow2.0\n",
    "\n",
    "\n",
    "内容包含：\n",
    "1. softmax回归的基本概念\n",
    "2. 如何获取 Fashion-MNIST 数据集和读取数据\n",
    "3. softmax回归模型的从零开始实现，实现一个对 Fashion-MNIST 训练集中的图像数据进行分类的模型\n",
    "4. 使用 Tensorflow 重新实现 softmax 回归模型\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_xq1py16",
    "id": "2031A689D83B4282ABA8CC2238D7FC9B",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## softmax 回归的基本概念\n",
    "\n",
    "- 分类问题  \n",
    "一个简单的图像分类问题，输入图像的高和宽均为2像素，色彩为灰度。  \n",
    "图像中的4像素分别记为$x_1, x_2, x_3, x_4$。  \n",
    "假设真实标签为狗、猫或者鸡，这些标签对应的离散值为$y_1, y_2, y_3$。  \n",
    "我们通常使用离散的数值来表示类别，例如$y_1=1, y_2=2, y_3=3$。\n",
    "\n",
    "- 权重矢量  \n",
    "$$\n",
    " \\begin{aligned} o_1 &= x_1 w_{11} + x_2 w_{21} + x_3 w_{31} + x_4 w_{41} + b_1 \\end{aligned} \n",
    "$$\n",
    "\n",
    "$$\n",
    " \\begin{aligned} o_2 &= x_1 w_{12} + x_2 w_{22} + x_3 w_{32} + x_4 w_{42} + b_2 \\end{aligned} \n",
    "$$\n",
    "\n",
    "$$\n",
    " \\begin{aligned} o_3 &= x_1 w_{13} + x_2 w_{23} + x_3 w_{33} + x_4 w_{43} + b_3 \\end{aligned} \n",
    "$$\n",
    "\n",
    "- 神经网络图  \n",
    "下图用神经网络图描绘了上面的计算。softmax回归同线性回归一样，也是一个单层神经网络。由于每个输出$o_1, o_2, o_3$的计算都要依赖于所有的输入$x_1, x_2, x_3, x_4$，softmax回归的输出层也是一个全连接层。\n",
    "\n",
    "![Image Name](../img/chapter03/3.8_mlp.svg)\n",
    "\n",
    "$$\n",
    "\\begin{aligned}softmax回归是一个单层神经网络\\end{aligned}\n",
    "$$\n",
    "\n",
    "既然分类问题需要得到离散的预测输出，一个简单的办法是将输出值$o_i$当作预测类别是$i$的置信度，并将值最大的输出所对应的类作为预测输出，即输出 $\\underset{i}{\\arg\\max} o_i$。例如，如果$o_1,o_2,o_3$分别为$0.1,10,0.1$，由于$o_2$最大，那么预测类别为2，其代表猫。\n",
    "\n",
    "- 输出问题  \n",
    "直接使用输出层的输出有两个问题：\n",
    "    1. 一方面，由于输出层的输出值的范围不确定，我们难以直观上判断这些值的意义。例如，刚才举的例子中的输出值10表示“很置信”图像类别为猫，因为该输出值是其他两类的输出值的100倍。但如果$o_1=o_3=10^3$，那么输出值10却又表示图像类别为猫的概率很低。\n",
    "    2. 另一方面，由于真实标签是离散值，这些离散值与不确定范围的输出值之间的误差难以衡量。\n",
    "\n",
    "softmax运算符（softmax operator）解决了以上两个问题。它通过下式将输出值变换成值为正且和为1的概率分布：\n",
    "\n",
    "$$\n",
    " \\hat{y}_1, \\hat{y}_2, \\hat{y}_3 = \\text{softmax}(o_1, o_2, o_3) \n",
    "$$\n",
    "\n",
    "其中\n",
    "\n",
    "$$\n",
    " \\hat{y}1 = \\frac{ \\exp(o_1)}{\\sum_{i=1}^3 \\exp(o_i)},\\quad \\hat{y}2 = \\frac{ \\exp(o_2)}{\\sum_{i=1}^3 \\exp(o_i)},\\quad \\hat{y}3 = \\frac{ \\exp(o_3)}{\\sum_{i=1}^3 \\exp(o_i)}. \n",
    "$$\n",
    "\n",
    "容易看出$\\hat{y}_1 + \\hat{y}_2 + \\hat{y}_3 = 1$且$0 \\leq \\hat{y}_1, \\hat{y}_2, \\hat{y}_3 \\leq 1$，因此$\\hat{y}_1, \\hat{y}_2, \\hat{y}_3$是一个合法的概率分布。这时候，如果$\\hat{y}_2=0.8$，不管$\\hat{y}_1$和$\\hat{y}_3$的值是多少，我们都知道图像类别为猫的概率是80%。此外，我们注意到\n",
    "\n",
    "$$\n",
    " \\underset{i}{\\arg\\max} o_i = \\underset{i}{\\arg\\max} \\hat{y}_i \n",
    "$$\n",
    "\n",
    "因此softmax运算不改变预测类别输出。\n",
    "\n",
    "- 计算效率\n",
    "    - 单样本矢量计算表达式  \n",
    "    为了提高计算效率，我们可以将单样本分类通过矢量计算来表达。在上面的图像分类问题中，假设softmax回归的权重和偏差参数分别为\n",
    "   \n",
    "$$\n",
    " \\boldsymbol{W} = \\begin{bmatrix} w_{11} & w_{12} & w_{13} \\\\ w_{21} & w_{22} & w_{23} \\\\ w_{31} & w_{32} & w_{33} \\\\ w_{41} & w_{42} & w_{43} \\end{bmatrix},\\quad \\boldsymbol{b} = \\begin{bmatrix} b_1 & b_2 & b_3 \\end{bmatrix}, \n",
    "$$\n",
    "\n",
    "设高和宽分别为2个像素的图像样本$i$的特征为\n",
    "   \n",
    "$$\n",
    "\\boldsymbol{x}^{(i)} = \\begin{bmatrix}x_1^{(i)} & x_2^{(i)} & x_3^{(i)} & x_4^{(i)}\\end{bmatrix},\n",
    "$$\n",
    "\n",
    "输出层的输出为\n",
    "\n",
    "$$\n",
    "\\boldsymbol{o}^{(i)} = \\begin{bmatrix}o_1^{(i)} & o_2^{(i)} & o_3^{(i)}\\end{bmatrix},\n",
    "$$\n",
    "\n",
    "预测为狗、猫或鸡的概率分布为\n",
    "    \n",
    "$$\n",
    "\\boldsymbol{\\hat{y}}^{(i)} = \\begin{bmatrix}\\hat{y}_1^{(i)} & \\hat{y}_2^{(i)} & \\hat{y}_3^{(i)}\\end{bmatrix}.\n",
    "$$\n",
    "\n",
    "softmax回归对样本$i$分类的矢量计算表达式为\n",
    "   \n",
    "$$\n",
    " \\begin{aligned} \\boldsymbol{o}^{(i)} &= \\boldsymbol{x}^{(i)} \\boldsymbol{W} + \\boldsymbol{b},\\\\ \\boldsymbol{\\hat{y}}^{(i)} &= \\text{softmax}(\\boldsymbol{o}^{(i)}). \\end{aligned} \n",
    "$$\n",
    "\n",
    "- 小批量矢量计算表达式  \n",
    "    为了进一步提升计算效率，我们通常对小批量数据做矢量计算。广义上讲，给定一个小批量样本，其批量大小为$n$，输入个数（特征数）为$d$，输出个数（类别数）为$q$。设批量特征为$\\boldsymbol{X} \\in \\mathbb{R}^{n \\times d}$。假设softmax回归的权重和偏差参数分别为$\\boldsymbol{W} \\in \\mathbb{R}^{d \\times q}$和$\\boldsymbol{b} \\in \\mathbb{R}^{1 \\times q}$。softmax回归的矢量计算表达式为\n",
    "\n",
    "$$\n",
    " \\begin{aligned} \\boldsymbol{O} &= \\boldsymbol{X} \\boldsymbol{W} + \\boldsymbol{b},\\\\ \\boldsymbol{\\hat{Y}} &= \\text{softmax}(\\boldsymbol{O}), \\end{aligned} \n",
    "$$\n",
    "\n",
    "其中的加法运算使用了广播机制，$\\boldsymbol{O}, \\boldsymbol{\\hat{Y}} \\in \\mathbb{R}^{n \\times q}$且这两个矩阵的第$i$行分别为样本$i$的输出$\\boldsymbol{o}^{(i)}$和概率分布$\\boldsymbol{\\hat{y}}^{(i)}$。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "900A996DF3114CBE8545153F973B4640",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 交叉熵损失函数\n",
    "\n",
    "对于样本$i$，我们构造向量$\\boldsymbol{y}^{(i)}\\in \\mathbb{R}^{q}$ ，使其第$y^{(i)}$（样本$i$类别的离散数值）个元素为1，其余为0。这样我们的训练目标可以设为使预测概率分布$\\boldsymbol{\\hat y}^{(i)}$尽可能接近真实的标签概率分布$\\boldsymbol{y}^{(i)}$。\n",
    "\n",
    "- 平方损失估计  \n",
    "\n",
    "$$\n",
    "\\begin{aligned}Loss = |\\boldsymbol{\\hat y}^{(i)}-\\boldsymbol{y}^{(i)}|^2/2\\end{aligned}\n",
    "$$\n",
    "  \n",
    "\n",
    "然而，想要预测分类结果正确，我们其实并不需要预测概率完全等于标签概率。例如，在图像分类的例子里，如果$y^{(i)}=3$，那么我们只需要$\\hat{y}^{(i)}_3$比其他两个预测值$\\hat{y}^{(i)}_1$和$\\hat{y}^{(i)}_2$大就行了。即使$\\hat{y}^{(i)}_3$值为0.6，不管其他两个预测值为多少，类别预测均正确。而平方损失则过于严格，例如$\\hat y^{(i)}_1=\\hat y^{(i)}_2=0.2$比$\\hat y^{(i)}_1=0, \\hat y^{(i)}_2=0.4$的损失要小很多，虽然两者都有同样正确的分类预测结果。\n",
    "\n",
    "改善上述问题的一个方法是使用更适合衡量两个概率分布差异的测量函数。其中，交叉熵（cross entropy）是一个常用的衡量方法：\n",
    "\n",
    "\n",
    "$$\n",
    "H\\left(\\boldsymbol y^{(i)}, \\boldsymbol {\\hat y}^{(i)}\\right ) = -\\sum_{j=1}^q y_j^{(i)} \\log \\hat y_j^{(i)},\n",
    "$$\n",
    "\n",
    "\n",
    "其中带下标的$y_j^{(i)}$是向量$\\boldsymbol y^{(i)}$中非0即1的元素，需要注意将它与样本$i$类别的离散数值，即不带下标的$y^{(i)}$区分。在上式中，我们知道向量$\\boldsymbol y^{(i)}$中只有第$y^{(i)}$个元素$y^{(i)}{y^{(i)}}$为1，其余全为0，于是$H(\\boldsymbol y^{(i)}, \\boldsymbol {\\hat y}^{(i)}) = -\\log \\hat y_{y^{(i)}}^{(i)}$。也就是说，**交叉熵只关心对正确类别的预测概率**，因为只要其值足够大，就可以确保分类结果正确。当然，遇到一个样本有多个标签时，例如图像里含有不止一个物体时，我们并不能做这一步简化。但即便对于这种情况，交叉熵同样只关心对图像中出现的物体类别的预测概率。\n",
    "\n",
    "假设训练数据集的样本数为$n$，交叉熵损失函数定义为 \n",
    "$$\n",
    "\\ell(\\boldsymbol{\\Theta}) = \\frac{1}{n} \\sum_{i=1}^n H\\left(\\boldsymbol y^{(i)}, \\boldsymbol {\\hat y}^{(i)}\\right ),\n",
    "$$\n",
    "\n",
    "\n",
    "其中$\\boldsymbol{\\Theta}$代表模型参数。同样地，如果每个样本只有一个标签，那么交叉熵损失可以简写成$\\ell(\\boldsymbol{\\Theta}) = -(1/n) \\sum_{i=1}^n \\log \\hat y_{y^{(i)}}^{(i)}$。从另一个角度来看，我们知道最小化$\\ell(\\boldsymbol{\\Theta})$等价于最大化$\\exp(-n\\ell(\\boldsymbol{\\Theta}))=\\prod_{i=1}^n \\hat y_{y^{(i)}}^{(i)}$，即**最小化交叉熵损失函数等价于最大化训练数据集所有标签类别的联合预测概率**。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9221297B8C694D4F9B76FDFD29C02E5E",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "在训练好softmax回归模型后，给定任一样本特征，就可以预测每个输出类别的概率。通常，我们把预测概率最大的类别作为输出类别。如果它与真实类别（标签）一致，说明这次预测是正确的。在后续实验中，我们将使用准确率（accuracy）来评价模型的表现。它等于正确预测数量与总预测数量之比。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_co5e0yp",
    "id": "3D1A30145B5A450E819DCA2FB094117B",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 获取Fashion-MNIST训练集和读取数据\n",
    "在介绍softmax回归的实现前我们先引入一个多类图像分类数据集。它将在后面的章节中被多次使用，以方便我们观察比较算法之间在模型精度和计算效率上的区别。图像分类数据集中最常用的是手写数字识别数据集MNIST[1]。但大部分模型在MNIST上的分类精度都超过了95%。为了更直观地观察算法之间的差异，我们将使用一个图像内容更加复杂的数据集Fashion-MNIST[2]。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:13.128375Z",
     "start_time": "2020-04-06T07:13:12.916911Z"
    },
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "graffitiCellId": "id_my8ejol",
    "id": "9C804F2DABDA482BB4375CB0D36E011D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "# import needed package\n",
    "%matplotlib inline\n",
    "from IPython import display\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import time\n",
    "import sys \n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_75u5u2k",
    "id": "810E6357484047F08D83319B46F7438F",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "下面，我们通过keras的dataset包来下载这个数据集。第一次调用时会自动从网上获取数据。我们通过参数train来指定获取训练数据集或测试数据集（testing data set）。测试数据集也叫测试集（testing set），只用来评价模型的表现，并不用来训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:12.904942Z",
     "start_time": "2020-04-06T07:13:09.398706Z"
    },
    "graffitiCellId": "id_no4tgy0",
    "id": "6FF62DC73B1D46128E613FC7DFFB3C3C",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import fashion_mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:47:23.855280Z",
     "start_time": "2020-03-29T08:47:23.851260Z"
    },
    "graffitiCellId": "id_d4dkkil",
    "id": "B4B6EB3C4CF74C1D8034A9A349870804",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10000)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 训练集中和测试集中的每个类别的图像数分别为6,000和1,000, 类别数为 10\n",
    "len(x_train), len(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:47:24.882399Z",
     "start_time": "2020-03-29T08:47:24.878382Z"
    },
    "graffitiCellId": "id_yndoddn",
    "id": "E35B37B4A94A44AA89722155E8AB5788",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28) uint8 <class 'numpy.ndarray'>\n",
      "9 uint8 <class 'numpy.uint8'>\n"
     ]
    }
   ],
   "source": [
    "# 我们可以通过下标来访问任意一个样本\n",
    "feature, label = x_train[0], y_train[0]\n",
    "# 28*28 像素，每个像素的数值为0到255之间8位无符号整数（uint8）\n",
    "print(feature.shape, feature.dtype, type(feature))\n",
    "print(label, label.dtype, type(label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fashion-MNIST中一共包括了10个类别，分别为t-shirt（T恤）、trouser（裤子）、pullover（套衫）、dress（连衣裙）、coat（外套）、sandal（凉鞋）、shirt（衬衫）、sneaker（运动鞋）、bag（包）和ankle boot（短靴）。以下函数可以将数值标签转成相应的文本标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T07:51:52.779134Z",
     "start_time": "2020-03-29T07:51:52.775997Z"
    },
    "graffitiCellId": "id_pku7gp5",
    "id": "B7454582B1484EE4B227BF145B0ED3BA",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_fashion_mnist_labels(labels):\n",
    "    text_labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',\n",
    "                   'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']\n",
    "    return [text_labels[int(i)] for i in labels]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:22:19.915810Z",
     "start_time": "2020-03-29T08:22:19.910824Z"
    },
    "graffitiCellId": "id_1x2gagm",
    "id": "D241CBCCD1CD47F28B9480080B93D4FD",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 定义一个可以在一行里画出多张图像和对应标签的函数。\n",
    "def show_fashion_mnist(images, labels):\n",
    "    # 这⾥的_表示我们忽略（不使⽤）的变量\n",
    "    _, figs = plt.subplots(1, len(images), figsize=(12, 12)) # 这里注意subplot 和subplots 的区别\n",
    "    for f, img, lbl in zip(figs, images, labels):\n",
    "        f.imshow(tf.reshape(img, shape=(28, 28)).numpy())\n",
    "        f.set_title(lbl)\n",
    "        f.axes.get_xaxis().set_visible(False)\n",
    "        f.axes.get_yaxis().set_visible(False)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:22:23.310144Z",
     "start_time": "2020-03-29T08:22:23.038834Z"
    },
    "graffitiCellId": "id_us9fuso",
    "id": "056F457B00454FFD81A3CB6AD966C508",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq8AAABaCAYAAACWob8eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO1deXydRbl+5uw5WZs0TZuudAMKyFqK7KsVUBEBQfSyiAroVVFU3OUq6pUruIEgKiIoqIAroiBgWQVKy9aF7mnTJW3TNHty1u/+8bzvWb6ck/WkOYF5fr/8Tr59Zr6Z+Waeed73NY7jwMLCwsLCwsLCwmI8wDPWCbCwsLCwsLCwsLAYLOzg1cLCwsLCwsLCYtzADl4tLCwsLCwsLCzGDezg1cLCwsLCwsLCYtzADl4tLCwsLCwsLCzGDezg1cLCwsLCwsLCYtxgVAevxpglxpiPDPXYAPecZYxxjDG+kaew+GGMucwY80w/x/9hjLl0X6ZpLPFWLA9jzF3GmBvGOh3jBW/1OmKMOdkYs3Ws0zReId+XuWOdjkLC1o/CYijjkLdCeY9FeVjmNQOj2WkZYxqMMacX+r6O45zpOM6v+3luvx/ysYItD4uBYOuIhYWFhUUu2MHrmxhvFXZ6sHizlcebLT/FAFumw4Mtt74wxnjHOg3FAls/LAqNAQevxpgvGmM2GGM6jDGrjDHnZhy7zBjzjDHm+8aYvcaYTcaYM/PcZ4ox5jVjzOfyHP+wMWa13OcRY8zMAZL2YWPMdmPMDmPMtRn3CRpjfijHtsv/wYzjHzXGrDfGtBhj/mqMqZf9T8kprxpjOo0xFw5UNoOFMeYeADMA/E3u/YUc51xmjNko5bzJGPNB1/GcZZwpv5B7PGuM+YExpgXA7wHcDuDt8tzWQuVpJLDl0T+MMYcbY5ZL3n8PICT7TzbGbDXGXGeMaQLwK9n/LmPMK8aYVmPMc8aYt2Xc6zpjzDa51xpjzGmy/2hjzEvGmHZjzE5jzM1jkdd8eKvWEUO2+UvS1+41xvzKGBMyOdhgM8iVImPMgZLnVmPMSmPMe2T/McaYJpMxyDLGnGuMeU3+95h0/7/HGPMHY0y1HNNlwiuMMVsAPFHQgshOf586bIy5XtJzt+xfaYw5KuOaemPMg8aY3fL+P5Vx7GhjzH+kPHYYY24xxgTyPPt4Y0yjMeYU2T7AGPMvw+/HGmPM+zPOvcsYc5sx5mFjTBeAU0ahLGz9GATy1Jl+37uk9ypjzDop21uNMUaOeaU/aTbGbARwtut5lxuOXzoM+6Qr92V+B8Kbsjwcx+n3D8AFAOrBge6FALoATJFjlwGIAfgoAC+AqwFsB2Dk+BIAHwEwC8BaAB/LuO8SAB+R/98LYD2AAwH4AHwVwHN50jMLgAPgPgClAA4BsBvA6XL8mwCeBzAJQC2A5wB8S46dCqAZwBEAggB+AuCpjHs7AOYOVCbD+QPQoGnMcawUQDuA/WV7CoCDhlLGGefGAXxSyrFE9j0zGnmy5TEq5RIAsBnAZwD4AZwv+b0BwMmSn+9J/S2RurwLwCIpj0ulbIMA9gfQCKA+o+3Mkf//A+C/5P8yAMeMdd5tHUnleQWA6QCqATwr775PmpDRXwG4C8AN8v/JALbK/36wb/2y1K1TAXRklNsGAGdk3PN+AF+U/68B+9JpUp9+BuC+jLrkALhb3kXJKJVHzjoM4HoAvQDOknf8XQDPyzkeAMsAfF3yPBvARgCL5fiRAI6R9z0LwGoA17jLFcBiefbRGXWuEcDlcu0R4PdE691dANoAHCdpCNn6Mbr1Y4h1ZjDv/SEAVeDEeTeAd8qxqwC8kVHu/5bzfXL8bHmGAXASgG4AR7jLeyz+3qzlMZyCeAXAOfL/ZQDWZxwLSwYmy/YSADeDDe4DrvssQfoD8g8AV2Qc80hmZ+Z4vjaKAzL23QjglxmN7ayMY4sBNMj/vwRwY8axMvAjN8vd2EehAjWg/w9xK4Dz4GrkgyzjzA/xlhzXF91gzZZH3nI5ERkDLdn3HNKD1ygyPooAboNMzjL2rQE7jLngwPZ0AH7XOU8B+B8AE8c6z7aO9MnzVRnbZ4F9Wp80YXCDkxMANAHwZFx3H4Dr5f8bANwp/5eD5MRM2V4N4LSM66aA/aV+7BwAs0e5PHLWYXDw+ljG9gIAPfL/ohzv9EsAfpXnGdcA+JOrXL8ETiIPydh/IYCnXdf+DMA3Mt7B3bZ+7Lv6MZQ6M8j3fnzG9h+QHqg/4Sr3dyBjsJbj3n8G8Gl3eY/F35u1PAYjG7jEpJckWwEcDGBixilN+o/jON3yb1nG8Q8C2AbggX4eMxPAjzKe0QKO2Kf2c01jxv+bQXYY8rt5MMccx+kEsGeA5xQcxpjbDZckO40xX3YcpwvsGK8CsMMY83djzAEZlwxUxplozLO/aGHLIwv1ALY50soFmfV5t+M4vRnbMwFcq21H2s90cJa9HuyQrgewyxjzOyMyGQBXAJgP4A1jzFJjzLtGK0OFwFusjuTr24aDegCNjuMkXffUPu9eAO8zlFa9D8Byx3G0vs0E8KeMerUaQAJAXZ60FhwD1OGmjFO7AYQMtZUzAdS72sSXNd3GmPnGmIdkSbwdwHeQ/U2DPPMPjuO8nrFvJoBFrvt+EMDkjHP2RV2y9aMf5Kszg3zv7jqlfUY9+pZ7CsaYM40xzxvKSVrBSYX73mOCN2t59Dt4NdSd/hzAfwOocRynClyyMEN4xvXg0sq9Jr+AvRHAlY7jVGX8lTiO81w/952e8f8MkK2C/M4czDFjTCmAGnBwPdpIDUYcx7nKcZwy+fuO7HvEcZwzwNnrG2C5j+g5ebaLBbY8cmMHgKmqLRLMyPjfnf5GAN92tZ2w4zj3AYDjOPc6jnM8WO8dUHIAx3HWOY7zAVBe8z0AD0h7KCa8VetIrr6tC2SQAQDGmMnui/JgO4DpxpjMvn4GpM9zHGcV+OE5E8DF4GBF0QjgTFfdCjmOk9lfjnpZ5avD/aARwCZXussdxzlLjt8G1pd5juNUgANb9zftAgDvNcZc47rvk677ljmOc3VmcoeZzaHA1o8BkKfODOa958MO9C13ALSzAfAggO8DqJNx0sNDuPeo481YHgMxr6VgRndLoi4HmdehIAZ2BKUA7nE1EsXtAL5kjDlInlNpjLlggPt+zRgTlmsuBw0tAC55fNUYU2uMmQjqnn4jx+4FcLkx5jAp4O8AeMFxnAY5vhPUR40G8t7bGFNnjHmPDB4iADrBGWyhnjvN5DFIGEPY8siN/4AazE8ZY3zGmPcBOLqf838O4CpjzCJDlBpjzjbGlBtj9jfGnCp1vRdAD6QcjTEfMsbUCuOiRkmFKuNC4a1aRz5hjJlmaPzyZbBvexXAQdJ3hUBSYDB4ARzYfMEY4zfGnAzg3QB+l3HOvQA+BUpW7s/YfzuAbwuJAelTzxl+toaO/upwP3gRQLuhkUqJoXHJwcaYhXK8HNRLdwpbf3WOe2wHcBrYDj8u+x4CMN8Y819Sln5jzEJjzIEjzecQYetHP+inzgzmvefDH8C6MM0YMwHAFzOOBUDN724AcUPj0HcUICsFwZu1PPodvMqs6ybwg7oTNI56dqgPcRwnCi45TAJwp3sA6zjOn8CZwO+Evl4BzvT6w5Og0PxxAN93HOdR2X8DgJcAvAbgdQDLZR8cx3kcwNfAWcEOUFB8UcY9rwfwa1kGeT8Ki++Cg+pW09fjggfAtWCH2QLqFT+OwuAJACsBNBljmgt0z0LAlkcOZLSVywDsBZfG/9jP+S+Bhkm3yPnr5VqAHcj/gisfTWD7+7IceyeAlcaYTgA/AnCRS45QDHir1pF7ATwKGhltBLWKa0Fj1McArAMwKD+0Up/eA/anzQB+CuASx3HeyDjtPlCH9oTjOJn5/RGAvwJ41BjTARrnLBp+toaF/upwTjiOkwAHYIcB2CTX/gJApZzyOZBF7AAnf7/PcRs4jrMFHMBeZ4z5iOM4HeBH+CKw3jUhbTy5L2HrR//IV2cG9d7z4OcAHgEnCcuR0SdLvfgUOKDbK8/460gzUUC8KctDrW8tLCwsLMYYxpgG0JjssbFOi0XxwdYPCwvCBimwsLCwsLCwsLAYN7CDVwsLCwsLCwsLi3EDKxuwsLCwsLCwsLAYN7DMq4WFhYWFhYWFxbiBHbxaWFhYWFhYWFiMG/iGcnLABJ0Qis2PeeHQiy5EncigHekWvDxKSwAAvulRAEBPa4jb3ZR2mGSGxEP+jYc5/zCVcW5H+UpD2yM8LR4fUZI6sLfZcZzawZxbqPIwAT8AIFJLN5vB3TEAgBONDv4mZSzLeAnLx9csQZZGKJPZl+VhSvj+oxWM7eGrYDnEErLdwrx5O+nhKhlOe+yJVvG3uqyL1yR5TVcry8Xf1DXsdGViLOpHIWCCrFtOZAh1ahAYy/YSq2SeQjWsD1GpJ7FeHs9yF+/lRlWY7aK1m/7tQ4281klmBlwaPsZr/RgtFEN5JKv4rr09dJfrRCJ5z9U+KFYqfU5zYfoNRTGURzFhKOUBvPnLpL8x2ZAGryGUYpE5rTCpKkK84Dw+pPPzlocGR8ozUEqcfAQAYMOFLP7/OYUu0nodfnxm+XcDACZ5OwEAhwVDA6bll20MqBJz2Ml8tJKR256NcIBz9csfBABMvZkfMvPsKwPeEwAecx7YPPBZxEjrh3fCBABA40fo8/uKyx4GAOyNs3G+3sYoiF2xoPymfchPLm0HAFT6+fE9Y8JKAMCXnj4PAFD9IvM98Y7/DDt9wOiWR/vFxwAApl69HgCwN8KPTKmfA6z2COuB5vWTU+gt57gQ9z/YWZG6V1eSZfN02/4AgC2dLNvyAMvnpOp1AIAfLD0dADDvsmWDTmcm9mX9qHmWedi/bCcAYGXHFABA55WMOphYuSbvtd65+wEAzvsb3/9kP+My/H3vYQCAhjNYpxKtbcNOH1Cg8sjTf/imMWLn6i9MAwC85zi+swk+DkB3Rvn+y318x1dV09Xnfv6+/Udnkuc83M0YEE+1MbJubaCDz+hkf/LS8/MBAPv/3yYAQLxp52CzB2Df1o/xgFEpD48Erkxmx27wzp8DAFh7JcdCj5z/fQDAHH+J6wb5oianEXE4ce5O8tpjf0G3yzP+J08QzDxpcsPWj2wMpTyAN3+Z9Dcms7IBCwsLCwsLCwuLcYMhMa8Wg4SLMfFOrAEA9NzHGe7VMx8EAAQMZ6UNUTJHu4Q5WdFFhiUuLGqJh8zbvJI067E1Wg0gzbQmnWxm/Yu9kwAAE/1kbz9/0L8AAFV3kaX5xsp3AwAmv3f1sLI4Gkjs3QsACLSx/O77XwZZe/s1SwEAl01hcLcTQgzyMsGbCuWNldEeAEBDnOzctcsZXbj+EZZPdGByYczgOZRMc9f7yfotW02W0BOm5MN4WB5Oku94S5z16Std78u6TzyZnosmpD60tJO1TiR4LBnn78vL5gIA/FNYH9bewciZ8z+2tDCZGgUEvSyPRaUbAABnVrwKAJj8Dy57boyx/Xz4mcsAAH8/6ZbUtSFDFnJ3kgzrqgjb2MzQHgDAhtbiXXrT+nHWfcxDTRvZ0Y2d7Dd64iIbEJlAV5Ss+wMrDwcAhEtZPloHACAq8iK/n33QjGq2vS0+tp8yH6857QSW8e6FbEA7f/12puGXI1vBsCgA8rCbx77K78UVE34NAKj2sD7skNOW9LAe1HopAXg9whWt1b31qXucUsbvQr2PdW17vBwAUOclA7vsoz8EALx2KdNw9etc2Zt0zhvZaRokA1sUkBUP42WanURGmt2rqMa1kj2AHC1yFvvX4MPsX81RB/OyZSsHdf2Yw51fYNhpbvoz+7PaH5HF9/57OQDAE+b3PNndPaj7WObVwsLCwsLCwsJi3KC4mNd+tKLeGjKNexdTg1Vx7/M5rzU+shBObABDjCHOnEaCir/w3hfVkDl8oYNaJGVNS2Q225Ng2j2G5wdMPGv7ta7pqXv6TPZM1m9yz2x3RTljbo6ROVGG9lsH/QUAcOvR1ITixdeHnrFRQjLANPpaaTTy5K+OBgD4P8w8tiSYl2rRBAPA6t55AIC73qButO4ezura9pMy3l0YA5TRwNrPU5OYbPZm7VfGNRhk/YjHxQBH2NPNW8i8edrZjJOhdB6NsLROwJVv2Q8f751o5Gy39kAykG0fYvlV/sbVvooA61qp3YvWsByW98wCABwW2gIAOCHE9jLvUs7kb37hjNS1n5/8KADg9V62oVIPmcXXO6bKGa2jl/ChwtUX7f0u3/9/WtlvbGpnXxjyMb/apiPCvBrpL5RxjURYP9SYEwB8wriWh6l9VfY2kuA5qq/2eth/qO567oepK27/IxlaXS2x2IfQb5eLzTxwGd/d52teBAA808t3VOUlk5V02CdWebhK1Svfn5NKaB9xenhr6l7bpS61ina+TvrandL37pRHl3tYf15e+DsAwCn/OgcAEDhjc3YaB7ADKUr0l9YB8tF97iIAwJ6DWY69c9gWT/o625MHDQCA7afynQyWbSw48r0X9379zRw35bnWBLm6pYaAznG0K7jwF/8EAFxRSXubU77MuuL9t1w4RCNRy7xaWFhYWFhYWFiMGxQV85rSmoh7J89hC1LHVl/JGZ9MGuHvIhvn6+Fo3f/oS7zWzbi6dCwwnpznGZ8PGJlXqT6In3okAOCsGrKcy7tmAQDComENygMnBWg9fkap6IzEjY1f0tqRjMt1aWYu4jDfOvsoF11Td5IszcY4X+0/Ot7G/QmxzJfJUq9DpmXtR8iwzH9x+PksNPydzH/3ROauYjPzv/RrRwEAHp9OdrB3YnoWWNHA8pjczJl+d61ogbWGD9oB2r7HzLuZ1rZPsh7s3cPZubOL76a7TDIRz55rmqiwqxNZn7Ky2C5eJXpzz089cm2iguW1ext9a80vQsZVsW0ztb6l8zij1zq8J0m9qtf0Zp3//PaZqf/nT+c5j4jmVb0N1AVZ5rtHK9EjgG/2LADAITU7AACNXXxHYT/beETaeHWIrE1tCduJz7AtxB2++6iwqdFkuv+oCrAjnRKizjqSZFnq6k9EGs7OHtZFZWLrQtRArrn4UADApFvzWJtbjB5cTFfLh6lDvmnyrQCAf/ZQ++2HsOtGXOzJ90SZ+oT0GBvj4jorw5earuTpvoiwtMrAxuTL0y315q9dvMfvD7gXAHDOxdcCyFghLSbGNR/bKNv9uZds+vSxAIApz7DdbDulEgDwoUtpU/JsC1dHvjDtFwCA3+zm+UtW0OvL1utoa+B58uURZaFgyMWoZuw3PtcQ0ZvuQ0yAY4pkB/sE1Tcr49pzDsdoP/7hTwAA7Q773ttbudpV8nGer+sHyX5ctuWCZV4tLCwsLCwsLCzGDYqLeZVRvs58GhdXpY598O1PAwCe3U2/hJuD9EMoMh74Tufsc/5PtwEA4g3UweWbTalPUYhFYaK9PduJdwGw9VTOTGp8nK2qP0bVuoY8nBE3x8huXPRTzlZLt5M5Kd/MmUjndM5YyralZyaOhzMlT5TnJoKihxSH9rsOZ1l+8wO/BQAs66IFu7K+MYfHf3DKfQCA2zB3hLktHDxxfRHMY/fEbC1ouJl5LmtKv7CYBGvomMZ8qQTY6ClFNPF3Q1cNuo/hLP3oxbTYffFl6niN6FM9Yb67ZAvrg7KnTjPrmTfDl3OiROq9XOvrYPnEakQnKfNW9Wiw/zVsL8VsE1y+lixP6Ay2m6Qwi41RMrJtIfrHTR5/mFyRXl3ZlaBltUdYyVLDY5u7q+WM5lFL93ARn0QG7bhKsptPJOmLtUI8AdQHyR53iy6x2icBKaR/0bwqi6blBQBB6Xu8SMo1vqxrlImFuA5+pWOaPFv0kicL23LrCDNpMWi4v4+KpTfcBgBYFuH+2b4WAMCqKL+RHQ6/O6VG2z77iZDUi4DUgUQ/y1N6TBlY3db6UiHa1zdiXOH4z/dvBwCc/R/qGuObqIE1fgkOMpBNSjHg6ENS/zp+8VpzPOv92sO5ElFeRc33r/5Ef9lTlzBfN/6bK56x02ijE17IfHsiHAvoqnLylVWjl/6hIA8z3oeFztjuE9xC9M3e/TmWuPcnNwMANsa5ah6SFYC7vk1PR5XrhJUfph7aMq8WFhYWFhYWFhbjBkXFvCZ7szVr0cPT1uTnV5KdUrbySQ9nfNueoPVw4m08d/PNZDGTL5PFqlnB2UDFy9SNNZ9IvcXuIznKr5PB/4THNsC0FLY43nXmCwCALtHZadpVqzZRfOit66kDANTfSIal40JqOnceTVp5yk3cv+2Lx6buPfF10TFNFO8KXs5ewk2c+c38BkWsvRfyuDKuE/3ity9GVvvqKvqZu/1IzpBTfufGEMoqG5mJeYQOVMleb9Ug5lxKIshkLukrYtGrYMY3+Z7f+0GyFK/Wsa727mE9SHRLiMZuCXnbmZ0nZVkBwNcl2m6p0km/lGWnaIErOIOufZQMQqJ5TwFzMjoo28o2r+1JGcVyL/uNf/fQG8FDv/85AGBjLJa69p9d1L/q7F8Zo22d1KxVFCHzuvtwslia5mMr6d9W8+0XJq05Tnr0GdHbvbqFLKl3i4SX7mI98WYQJf4uqQ9SRIkgz2k9iPf89En0zqC+p+eX7gIAzAiwnJ4OzylADi2GAjcLFn9sBgBgdZT9RkOMTOt7S8nIrxJyU5l4d2DXgDN0DyzKuOqv6s61jm6JcyVjV2I7AGDHO+k7tva2zZKHdJscM+Rh+LwVrOtti+mHtHRbejzia2Hp1d3F72bsk+wvdzRxBXfe1+n32DeT45G4PCP0MiPTmaO4arJlMVlIbYtTBxfocvQxgD9e3yzWtfikytS+SC37l51HiU5+koQbFpudV8V//VMdzPv8UBMAoOYZWR0faZJHeL2FhYWFhYWFhYXFPkNxMK8uzUPn+8k8XrJgSeqUDTGyKtMC1PNcUC+x2D/E31vWnAQA6NrImYGnlPdqOobj823n8HonxvH+hOWi8bqUUavao7OReDxYwEwBX5pEne5DojdVndkEf/aMd3YJbZ1XgNq9p2/+KdOcoFbppPmfAQBsevdPU9ec+Pq5AIB/HfR7AEBYvA18Y/dBAIDnD+VsqFtYKi03nSnHxJr4LxLNa8cJLLfJwwtxX1BEy1gfJOnw9op+Ux1GSPE5GVJYx20s6cn+TfQN7140cOvA7jmTdRnfyz7PK4yr6nlV1+rtEe1rRnnoMY/oYB33NFW2q+4eP5GSyraSCWlN0rJZ2VNllnYJA/njvVzJUB+UQJqtXNtLdkp16B5TvGLo2tv4bu5+7BQAwPrLma/ggbR0nvodse5dqj6a2Y/MlV9lkkw52R6nNB3TPlkhbH4J+wNfB6mgSbdSg/cPkGE68mWW8fGlawEA2ySC3en19Pe6zPIfY4bvznkwa7tKImZ5xauA9vUK1TyntK3y4x2CQYBeq/dSzbQ+q8rDb1aNh/Vr7+GywqM3KAKvAyntsEbQUsv6CfwG+uR703xoOoJj+wnM9/qTuapz7GevAgDM+122d5b45sasbWeatNkW8aBTz/uc+X627VefotcO89yrI8rTSGH8UiYRlolG9UvezJXaaeXsU7Z1p5nzT0x9EgDwWBvHHJ+upcPWj627GADwrzZGEasUnXxLgitJjn9ww86BPEDZnsfCwsLCwsLCwmLcYGyY11xxcjNwzHXUa55S1tcSb6rMErscslWtMpr/xoK/AwB2z6fmVa1nf7GOOtFOYWS9cT77mA/Tz9p51Yw1fOODh8DjuFVBw4NGlHghQqtxt0ZP9UGT/WRQXu6emXX9WeddBgDw9PC8GdOZ5rO+/o7UOeWGM9zzI4u5Q3SirafTurEcnBE+tZfbJ1eTKVGWSn93S8zq3reLvviHQ85uwaE6zRSbqsS8x7WdUY3cxzzx7P3JbIcFRQW35W18YwN/N9GDRmAm62W8V/wxqtZVGOiUltGTLhAxPEdvTbZuWKerwa3ZrMx4gH87LXvPK+Xv7W1kFrUOK4Ok+u5MdCQlYpQyRWJN3xtjZSsbrUSPAGtvp59EJcamPCkM0SvMd3QCK/lFq6lH1bxt6J0EAFjVTvZrWwdzF4mnG4EjekcjvnHrytn+r5hGbeIDu+ijevlHyBy90kaNq7OdK1VjFhFoMBjAZ6WbcQMGYQU/gCYwdR+NLhSN9nlGodEU5zetKkBWLM20qo9WvuOOJOtBuThJd9tgKIsazVi68ebxVJF5DpCOVLdHIm8p+7tDVg1/fTqZym/jMBQLcr1/AHA6mXb9ZnSemK7j9b9jmS2+mPnQ7+tAiNRxfBKtYJ2ctIxt9uEo+/bJIZZfaCq1wdiKMYHbc0DyVfqc913CdDVsEwf76Emdcyvmy3+sRx/H8QCAGzc9AACo9bINfH8XV47ue/hEAMB+68k6u9tjyqdspi/+fpqPZV4tLCwsLCwsLCzGDcaGeR1gNrquk8zBnoo0H9IUpwarRiJ86Cxylp/Wr7sTwr6InlRniP9z0N8AAL0HclaqM8ljQ7SGvGDVJQCAUmwcdnbc2Pl5zmImexm9p0EUP+o7sU4YV9XoafSr+GlHAAB6aiXaTbVERJEJddfktIWvWgmrPicRED98VWIFehVndseWUZeyKyZWwyF6XVCWqlJmypceSM8ITyKtixsr6MzX152tdU3pWFWylYsEcVUt79CCdhQVHI+8ozLW9T2i9UwEJQJbh+jPpH54MvLqJh/dZVWyq/i9L7ihviIVKW8Dom31uzKZyJibh8Wva9Cj0epYWK1tZEYmjkJ6R4qpj/EdbSdxgeZzWA9uPIpax2v//iEAwN1fpd/ESCXz2y7dRLxU45LLT4Y3Cke8T2iUtq4kWbz/+8NFAIBAB4/vvU5Yf/FOkmxl3/zFU9mv/uVU+rOM72gaQU4LjHzfF6MdSF8hXT7GdeuXuXL34yt+BgC4cc4hOc9L3WeIUYKGg+QJhwMAFgafAQCsEz+atV7qE9skImOtT1fXJOKWeKdQhtbrZPv4zfTzmtLHym/KL7SwucrMKour24dItMjWpPjoThbWjqQgyFM/EntoF1LyF6787veXvud4ysWbUWdn7nu57XfqWdbBvdwOtMpq6l9ZTj0zeL/IfGrxx4p5zYf4NsY4rDMAACAASURBVI6TdOXBZOhV89X1T6ym5vXJQ+lDfn0Hxz/z394AQHnavt4z3Nu+qfUwO/OvEFrm1cLCwsLCwsLCYtygOLwNuFAb5KxGtaEAEJBZ4/YYrV3X9TBW8Np2MgHvrKN/UtVyKrOobEy9nzq5lLW93Pe4OjKuhXS3Fn+RafzexDMBABdOoq52XoDatOlezlJ/JdZ4Gkf84btvlzwk5FctOfkbMhlMkof58Mj8I+IwR37D/KuPyztbjgMATA3ulXvoeSzPJ1vpg+3ZR8igzMTYxyp3W8Yn83gZ6GNBn3mN1GxvhPWgp3YcMI0uXV14h1j0HiQZl/ymImmpD9uAeB3ozYiwJd4VfLJPWdlotUQn25bNUo6nyDd7kz1Z28qwaix33Y5l6PP0f21ryhQlO4pX+3viV6gN60yQvVrWTB+Sd26ntuySU54CAHzj/dm2AZ1JMtEtSWXaxDdnhki8W9g2jbJUKaLoaT6yeCujLOOvbH4vAGBdM7np0GusWLds5P4pO8a+v8gLFwvWX93e9QkyrK2HsMy+f+rvAABNcfrzfKmbkR2b/0ad38R3r815H0+I5bPuW2RH53y+8N48kn7Wb/0eKEs6XSKvReQ96zew3NuTtR0wiaxtyLfAk9GhavvQc6IuglG/qyn9rHxXupI8sVfq1zvDTNMPhpXTsUEfbTQA483W+up2nwhULuh3x98pBahFLH7Zo+USDS8+etroESHVhlgfcrGt7m+Hcx+Z1uBh4slEfPKfX0df/feViyeDjo7sGx3DMUjdDxoAAK/urET0s/mNVSzzamFhYWFhYWFhMW4wpt4G3LMX7wQylidV0W/h7kRF6pLWBPV+VV5aAHbEOcNt6eH+A4LUci7vngUAqA3szTq/QaI9zAtSm3XjztMAANND1LnETzsRzguFmSVP+w7ZiLbvcPvOydSf9ryNzEnTx8iMXP826sZWdtKi76Y9ZGLXdZNNLvWqTm/gqCTqr1JnxHskxvTcMNneX6+n79xJ57zhupIsdzEwrr7JtGxOEWauKFn9Ma0KZWc1opZfNMGq//OUslySXYXxLDGaqGiQmb+822RA9NyUH6K0UXRo4kEjUp2evQdapY0JMeBV42dPdmSl8YhYPs0asn1PJjM0fBFZcdF2olo+b1fxzt/vf5SrJkceT08hn5/DqFefe/ECAMCGf5INvLuWVrylW8VKVwl86d3V56+Tg8QwUnfEFWOqXsTE3KB3OivO+jPvAABcXn8ynzmTrO/pyz4MAPAuWT6MHBYY7hjprnpiDqc/yg0X8bsy+6i0T84l+98EAPhNO5nVR1t5bmMXv0lnTuLK3h/edieAtGW1G9uvpN3CnCO2DD8fA2DXUWQ7yzz8Va2qX/LfJuyneiNQu5D2ZLaz61S0LJeuFUCqz/WqtwEo2xvL+lXoSl6dlyzcxghtJ7bEad8RXXwUACDwyEtDyutYIBebqvscYQtTlvEKd90TxMVV7LEXcG3334/TW8Hcu3ifQAfL19c10nhTo4RBeMvIZKgBoOoejqNe+xbHObNKuXqxtncKAGDvOWxb5ZvJ4l7xiz/LlbRpOCRIne0XPvgBbNmaPwpc8fbcFhYWFhYWFhYWFi6MqbeBlLZEZjWNV1ALcWqYjORzvVNTl9T6OFNR7dqUIGd05XUSdUeY2WqJnNOR4MxPrYr1+iMkNvdnHuMMufxgzgoq/B5glGSR8Sb6RvTL79Qe6qFCd4qfPXlwpY8sseZNLaNjOSgT1SR5ZIqs50z0M5/tceZf8x15sbpg+RktON2izVJZTb5JX679bl+wAtXLBtrFMn8cMK4Kf5dqnl0VU7W/Gj1MDHpNxiRVrVt7J/JaIeJT0Fj24xH+PH6ilXFV/5XIKI+0v8rsmOzJ2uLV+Jbszxj1e8W/79PCCpYuZdvuWcS6fPY8al5V++heqdG+IZnRODQ6mTLR2tfEkzxneQtXidof4KrQDQu5KvRiI31SH9JEi+Lpy9cDUM+iowSXFlx1pcne3uzzXCyRt44rWGu+z+/Ig8fTpmBbgozkkvYDU+d+YfupAIAy6XxqA/yO/HvjPABA90Qyimfd83kAwCxkx7LfdAl/X7qSjrLPO/tSAED0VPrL9T1RuNCF2h2ofYN6C+hI5u4w9V2rv1f12KMeebS9ZH5n3J47tM9VtrZGvqtvyHd3ho8rnUHDdqVa2GoP09b+37Sun/jIIDNZSORhRUcCHbO4GVg3axvi8AL/WrUAAFB3GFdC0crvcutc1tEpT3YWLG0FQb4yy+x7U947crOj/+igZw6NIHpIiCsdN9zIlfWEXPe8fO+1fl69hn1LycZNcJz8/bNlXi0sLCwsLCwsLMYNxoR5Ves098x54uscZTcnsuMkA2kLSZ0tHlu9CQCwWxjW5T37AUhbVtZ6OLOZ7ufU5/Vezowf7poLALjiXY8BAO674wze/5/PwTgFjhojsxSPRF1J5VdmMxujnHUFXAxrwjWnUJY1MRjRp8DNvgiZm05aP9FmxgqOk1+bN1QYuVeiCN0M5oUreo8nxve+aw81ep4o33+gNbseBEnQIRZLz4qFeEfJrmxvC75OLdz8WqJih9e1ROJ1+Z5U/V0X0i9f2aewiH/V/+S8abtGN7EjwIlT6QmlRNL8zsrXAAD/aWLkrfYe8QctfqK3dUtsdrHujcTZxv1e1itlVQHAEfrOCPM6MUQWtzvOex5URduApd1kXvcLspwWTOb+OWVcwVoxi15f8Fr7yDKbC2obIZHjlODpw7gKus5fBADY8V6W1z9OuAUAsLx3GgDg1l1kV3vk+zIrvCd17dvK6GBT/WE3Rfh7yQL6/Hxh7ywAwMXvpt/sxReTPWpKkHm+bcvJAIBzZ9C+wVu2DQAQauU7KaSi0e8i6bS+t4nD53aHdVvrfMDV1j0pTwKDX33xuKJ1hQ0/KKqFrfbye7M2xroWMDyvVTxelAfHcIVjFL9t+bwNJE/i6mrtyxxT1N2+AgDQ8qGFAICmc/ntl6YLrNk0amkcFgZTZgNEm3viEC73nb6CY7HTSnj+Ed+8GgAQk6hjP7mSKyLTffyQ7VlCbew09F8mlnm1sLCwsLCwsLAYNxgZ86ozYx9nfMYrY2GPWC/2ipjBNULP52/vRz/jTLlRomk1SVQXIO01QDU3z/dwRqt6nVofZ/7tyewIURrTPObS91xXsw4A8Me20wfK5fAhs5ekyzeafwVnFOu7aV1fIrPWvfFsYaJqYVXXmmueo2yT5k/vUebLfmag3R16Shi4AfzU7Uu49UPGpe3sc34GoZDvHEdYm5SEa5AxyscErrRFqlgeVZXUk7V0cztSzfajb9g0y0pGOF0g3gqek4y6aGzxNtAxg+1Ca9x48O+q8Lg0r6rP87gYJm/GdgwsB12R6BWWanEd9aKPoALFBp/4Xm2J8i2pTjfQzv3+EuYlLpU/IOcHvOK3U/oNvU/cpOuCsm9x6Tf8ck6ZRCjUcgrvzu4fDiinbj/FYM9guYVeG34+80JXYvL0UVu+Tt+s/30RbSROCP8IQFpr98Nd9CijTOuiiuwoiqoVBdJ6YK1LcRHLv9JG1nZG6d6sa7+4/jwAQPAdDbKHzO2G/yPz+ov3MSLX31ppXb7qCmoe8XK+zA4el378YQBpf75dSdoz1MhK5aEBrj7qN0F1zSOBrny2JLL9pFdrxEvRP25M0E2FRpfcLktfSw6mRfliQ0ayGFb6RgK3vY5i03f5/mMTWF4H3CLl8EmuloT2MN+T76cHkfgBMwDkX00oGuTQwKb8u8Zj2cfk3AcbqQtfLyuCi+u5MlKLbK9OrR+lblo9WMy6h14HBhqZWObVwsLCwsLCwsJi3MAOXi0sLCwsLCwsLMYNhiUbcFPmqbBgg3R83nMOKfTG95Ja/+DhFMU3xcsBAC9LoIFKbzoMZKm45tCls+1ROo9WGYC6yJok8gE1btom4WQVKj/YGheXWu+hmLjq7sGlfThwB2NItPPZ7bLEX+VnPrtFva1Lcrrsp/IBb8Y6uccV/jYhyzZ7xSvylECbnMdrTKL4l2lMqXh01tUH9TUuKxa69K8Sgf4MuxyVtDjZN/GUiJudYnSZ5ZIyhJtY53eurgEAVGxjHuJhCbsnK009kyTwQIZEILCFZalux2JsWihp4rnd9cVfH9wwR9K5daWHDr91WVQNHhWp8JcZ7cXrZIe7VIOto8JcSn4Eh41WsoeNlBzCqDs89rvBZr74UIkYesoSt8oDki7XarqdGbRBWYseMdCK+XkPlTCp0VdoK/vH5jjlARpeV11rRSvEef1wM5kDiVPoxnDLO/iOvHPZX5aI0c+hk+jEfGHoaQDAmu7JAIAnW+hKbD9xil4lrgfnlvB6NYTdIVE+yr3ppVqVUaicRMshJp1Nc4TL4S1RtquvzaFUwbuB72amfHce7uKz79lNSUNdkPvfuEqiPlw5lJLIjQvLxfhHqre6vlKjqT91MnhFvbiv8qZkZyN3j6fl1Cr1YJafQX40XLk+Iyh1NyzGZA92sg8b73IBRSqw0kE0WNz8TZZHiZdGR7Fmfts3XsTxR+V6eQcSjTq5Hw0hPREZQ412gl2BoWCyecuU4fYQ5HT5jL0Xvsw8XbLx3QCArhN357xeXd6pXOBPbXQrF9+6bVDPt8yrhYWFhYWFhYXFuMGwmNd8AnrfFM6AY/vREKnlQM5Suydz1H/YWasBAJfV/QpAOvyruvpojHF2dni4AQDwRNuC1L2bfZy5Kht7bCkNrlqTfIbOMq9bfz4AoC5MxuAXMylujwnzsibG2XybsBWfWvBvAMCfUDuovA8Hjtt5tMxuojJ7VWMBZUjcDqJjwgaEcsT09KTcA2WzLilWSvYbt2ekPA6txxQqCldtuCs8bPq8wd/ScRn3pGae4wDbTmLdLmvgdmWDsGI9ElKwlbRqvIp1urfan7pWAxx4Izy3c2oAmdg7icfVyXp8s4TKLGKDtpZD2F/8s5v57Uxw5l7u6ck6T2fybgMuIN1OWmTV47iguJU6iy5sgg8vLXSyR4wUyylt2reFbqvKQ6U5z1emVg25QsLI+jJMPpVZVXY6mlSDtuy+3YjRrfYzKTdc2nd5CxfsIjqlFI0fORZHnEUjuoODZBTV8E4Dr5SKMepOcWelaasv4WqTugRr7CXrtd5h3x4SZlKNsaoDadeIeo8Jfu5Tg7XaAH9r/FypUbZ2XYTfOl0JfD3F6LOdTZS0zwo1D6MkcsM7j4zqFB9XHpZFyETXy2qisqFRYei171f2OL2dHdyg1ESzjmdC8xd1fU9aJDjB/n4yyx3SX+yO0wXUPD/fRZe8i3cJG34HZg8534NFPiOq4d4nk53UFbtEu7iEO5pGgcn/JfPcvZHunSZP5Thk8kclDLuLlVQXWm3zOZ6Z8DhXfkbc27oNqvKESh5p2WTB9Y3oknDVf1zPujH9/BXZ57u+LSbAtlLvY13542qufs3GK4N6vGVeLSwsLCwsLCwsxg2GxbxGziRLMekrnDUcVkE3IQtKngGQ1g0pU7iqh+H5dFa6LspZa5voM3X2vytKYd5Nm+i+6vGjb08986vb3wkA8JRwBrFHXHKcV6bOsfnMK2c8BQCYHSA78VAXZ0TbRftaJzPCWX7qMN5XvhbA6DKv+XDyBLrLWCWOwJX1SLjctnj70Kb5odd0CCuljEIhHP+POnyDTKROZnOQPm6tqyPMUCr/AX/fi8Yarhmpd38G0ug5gCxPooFMY7SKaY9U8/zyjXzH6mGta2Z6JuxvY9OOlev81BU6s1Nc21xO5nXG9Y1ZaShGNJ9Mhkh1den2IXoy1TULU5fMMTdXRk31n7/tIFPU8jEyZVMeHpWkDwtu7apqFzXcdMg3I+s8ZRS1zUcSrAM+o+WRvl8ykV02vSLG86Uc2Eu/Uco6tlZ0paojVSQKKHYN7o5g1s/WYdtS1v+XjpO2egDfzWFTqYWbWUIWb0GY2teUPYR8d3Qlb2EZ68ciCUsZk/yHpL5UetL9TdgE5NrsPmiL2EY0yrdKV/o0/Kmumu0WTXClsKDbItTVquvC6f+Q+w2uKHKi6fS6rO1eYU6r5LvRFmf+mkXgfliILoc0aIF+V5RxdWtgh6KJ1XLYnuiVe7Lc9Lsbln54t+q0zej3u31YRXcY6UHqbXOFflXGVdnv9Z+VvuYZtsHaI8mwV5y5of+ba6AN/S61FSi4Rz7GNQ/MQjLHa67gasaCb7MtxRu3Zp/oDs1cml7tUbuRdT+hC6wzqukvr+Gd2Sth6QtcLlNFMxvS1eGGbDenMKZfMbBlXi0sLCwsLCwsLMYNhsa8Gs5GFn2HurDTylcCALplZqcz3+0uC/9Kma1HYnychuBTzA8y5OC5FdQ6PHULR/LH934ydc6GU6mTfbyHMwGd6V60iSH/lm8hg3TMLAYAOKScs3Rld1WrlAodKTPn53vLBp39YcPJzZyqnkhR6eOMRctRGVePzKI8GdOQlAcC1VrJzFeDE+yNMd/KDCT87lloEYYHVdZUJmhubwN9AhHkmJWlGFePK7+6WSN1s3kPigauGWnje8gGlohsKhFingIySe+ewXdXvo2/LQdIM854pWHxTNB6MK8N7dIAB+LIXkLM9tQLy3A4Lfmdl1eOPD+jhAsOXQYA6JCQ0MqiKkuYQHYgklwISPufKN5JWmQF57oDHwUA3I3phU72qKFSnNGrttWtS3Wv2GQxufJvIqW357mdcfaLGrQgUUpGcslmsqEXz38JANAm+lNn8GTd4OAxKHmB9gwzH8mOad0WZp/2zEFc+dt7AN9dx0wmoncK0+wEtQPRe0pHkZQQyXuEZe5KJz7YIr+t0l5aE7KfbL+3U7S/HdmskhMSLbmb5dtOBnJNK+tbifNi/jwPEu5qrauP/oB4p5EOc0EJv30aFrZDA1GYbO206ld1BaMrI8CP2+5Cz01KeIJW0bzuTpRnbR8alLC4wmB3Odla+32KoXo2cLGWufSh67/FMk80Mb/+g9gpTzh73eAeIbYmvRPFA8hIgxMYAxMMwkjanYSsmIgHKGVF7zn7NgDAk50HyoWsj7dWsk9dchLb9x8OnJx9f/02yf0zvfToCuFXTv8LAOCBi06RI7Rt8pSzbiQ7aH/Uh8Wt42q3arLrn3Ez5/1zq5Z5tbCwsLCwsLCwGDcYEvMam1SK7f91NK6v/AkA4N6WYwAA00Octs4MUPdxaMnmrOvKPZxd7F/BkfVDXQy5t6T1AADAFD99oz3dPQcA8Lvr/w8AcNlnrk3d4+0PXwUAaJ/F8Xa8lDOYikPJoH318L8DSM8QdSZYHeRMQf27KpQtVktl7/5zYRqeGVxBFAiqTVKtq2qCgzJDVh2RsqyZjFKbsE+qUwqLQ09lWpuS2ex2tKrQFEnh4QTJiKT8uLqTnM/7QD9I+bdVX5fh4PATuI/QdRDfZelKplVZ5IQmPaCMGgsql55ZZ/hG2CaRBaJkKhnHeAfrh6+dF3fMJaNQVoDwlaOF86q44vN6L9nRlI9j1xxcvQ0k8sUMRpq1VR+ZJ5XsAAD8Jky/jcnu7twX7kM09nCVYHKI7I4yZ4qaINPYEc/WX8ZdiyqpMNMZYUJTfqJdvmHV76se17oX2cr6ET6AjM5eh/1rIbX0TjyOxM5d8FYx9Ldv9qysNKTSvovfi5r11OdNFP/QTiQ7xLFRDb0ycOpvOyxC3QyNvfY9yQD3JcSXcrSCv/HJojsvp5ZVFsdSbKgY+iMeFj12B0O2emPyndokDNszD/RTAv2j7p+imP2mPBOqYRUftU625rdLtpWh1Xah14UlnKxqDjNXAnVFz83S6j0U+qywdDDlwvp3a2j0VBvcByt9LubUW8cVrOR0/nZNp14z/KcXcl/fD1O7/occ65gE8zl9AVeL0+GBXUnxZzPOKV/4PlklqCmQtx/HgROJ5P0kHnAw9d7HhWQFFrSzCYh/g2d7ZgEAjinhivUdl5wLAKi6OzuEa66ymfUbtr8bnj8bADD/1WVZx1OMax5EZrCNbJNV9KF6erHMq4WFhYWFhYWFxbjBkJhXTwwI70zioXbxx1VCi31lEB/ppAXbtBL6OlOfrHNF0/pKL2et/9xNfV19CRmFnTHOtPfEODPS6De//MHNqWfftJMeCM6tXg4AODRAxrVV/MitEg8GHUnOqnUW2ZZQzSvTopaWGnGnSmaf7YfUILFzWM4Xho1YHtpCtWpJ1/FMHZLHNddKunRvuq3a3rjLKriP79kigOPX6B+yrdkfRlI98eyLUqR1EU/XPAdzJcLbxFm7Mq3iYjLF7kCsiuMl2Zkx8TRDpXJHJ8XSsjB7e3jvZC0Zk2ATb9pdK5rpkWej4PBNppX1kcKKPdfNylwtrKl6GVCvA1r3M5mkVLQqYYCqvCzUL770PgDAn4+lJqznZPZNY+nvVSPPuP0+r49k69HU32lXPJvl0fyHfWR71J90JvOqUN+nek5C+lNlax0/t0u38LdMbAciQj0m3Vr6AiDRKlrX1racx1VLZ4KSb9UmVnG/UyJ1PJDdnzs+WakQJte4KWoAjpfnqHY+0MoyDDcIE6/aQumrHH2G3kueodueDl6XWL8pZ16Ggq3nz8za1tXF1iSfdbSwx8/2qt9XHlfWVL0y6IqE2la0SpvwZ3gbTbhW+5Sp0/36PVXbEz0vJOXT62Qzt/uEeXWxg91HsrzaZ4jHg3ZhZCuY5pTP1jxQzwIAcORR1LROEN/ADUfnsahPpUW0p4ls7bAWQ2xaFIVAsiqMnpOPTuVxyp2vcr9oU4+t2Zh1/hsRejZa0UUPUBo5bms5fex/5iu/AwD86u7suqaIPzYj9f8na3nu5q+SQR2qB9nIBLGRik/Iedx4TL/Vpog/5RYWFhYWFhYWFhbZGBLV6I0mUd4YSTECTzSTKaoLUdtwWDn1FRpr+vUejvKXiz9CjRddGeDsXZmDiX5ev1+QFpo6U1zamx7lX127BACwRUbpf+tiHGv1kTpBPBq83s7tbmEj1Ndhb5yscGWQz15YTV3uGtAP7O5DPYg/O5TSGDlSTKqLvMin1cvUvLktid2+YfUdKYsdDxcf0+qG6s7SO/iTYhGHMdVSskmZ13i5xEsfRvpGG11zyAikvCxI60wIwZTSvIqONelqvcmqdP3wiHYRPvV3K5ubyeo5s9lenN0SladSjkuUvPiOphHlpZBoO24WAMAr1qfdUhC1PvYbiRRDyfzXCiuUqXNP68flHtIujp9Nv4xhaTd7FrDc6sfQ36uT0gsyX9pvPrVnnpxBP6+qlVeWNO5qIB6X1jVztUbPjasvWI+u2AhzJqxctJLHq9cwDcrepZjZMZDSp7R0bkndIKtsf0k2ec4ZrvfjQnpN9p2WHa1LvW60iK3EfrL/mhs+AQD4q9iOVHp4fFM8O+JWq3gX0BWKzAhbyrAqix+VAqkRGxFtY/PDXC29fMsJAID3zngaALA6mlsz7pvFb3q8YZgeb43Jr011aV519cTtwX3Q7+SOtCeAD9VRA3rLZe/nowaIApXyVOCO8CgrnkfM4fijf1XowEgEDdpm+/DUtTcBAB77JFepNkcnAgBOL2O0ui2SHo1K+K4qpv8dYbbriMNf9cf7pdsvAADs/3MyuL3f5e+v5v0m9ez/Wn0JAKB0Wza7O1h0TmGfvL63LudxJ+lYP68WFhYWFhYWFhZvDgxN5NnZA8+TL+P+R48DAHztnPsBAE+K14CHmshutkeFGQlztF4hzGq1iPfU72tImBKNQhLxcNSvs76mSGXq0c8myTrEJIpMRH6VlWiRmYbGt+4QkWeDWH02t1Hb0Rtmlp9J0LPBOyfTr2XJLgNPAcP+ZmEAf3P5/FIqm+rWtwJpa2lF2qJYfT2qBanEey5GqtGFRFCtg/mTeh8ma/egoCSUkgkesfptnce6WbNk2MkcNSR92WyWkBsQggVJv3gQEBokRb5LwQRK0zqqFPMazfbnWrOcZVxzDDXj63fy5kl1wTdJ9EdFxLxuO5MvUWO5dwrzqmyqxnKf5SMzpTNy9SQCAJO87IPWRjnL7xDW6e2VZF7VT3LngsJo0QqBpGs15Y2dtJqeKcyre5VFNa4aJSvoFa8lyb6N362Nj6aic2W3st5KqS+vtMozec9Un1T8TkzeNCjxs+w3xaj1ni52H27bieo7yRIeu/CzAIBbzrgbADDbR69AhwXZfh7v4cur8fRlSaNQ/7/8bRdbkv3Ep2xEvmnX7jgCALDijoN54Q1kXmNw6875vdryfnoaqr9xmMxrf99S9zFhPYNL2OZPqKFu9d7bFgMAJt36XM7bbLiJngVWz781tW/+P67k77MvDT3NmUmUPn5+GVeZl42QP/Tv7MLkHzyHr1x8MgDgU5OeAAAcEqT3lF6pG0u6ZwEApkmdWRCgXdKyCFn5Wq/4CgZXVja95w4+4D38eTHC97czkfYFHP5Wtkcjtx/XgaCrfeu7lBtvyT5hgPtY5tXCwsLCwsLCwmLcYFjm9bOv48zup6+dz+2P03fYmZNXAACWt1PXskVYz1dF++oXXVXYz1G+WroGvGoJLFovmc6XetMsiOpj1W+rRszyuLSf6p/uxbZZAIC6MBmXuRVkZVTrpYzLnZuO5Xk/eQ4NThdGBXniDbcLOxwO5GZ7dEatzGwu62n3rFtnymp5rVbBfaNTFV+Erc7p2S4RUuypW/vajxeClCWxaIuUxVQWN9xcSBVaYdFTI2xHgGkXZx7Yu0DahUTa8nWIzz7RwmreKsvSTGMiwNUMTy/PVb+EzsNk73Z0SPQT8UbgVEl0In/xUfSzZ5GlmO1jRk8sZ3+jLOCrPbSMPVGqz6LrPg8AqLon7avwt40UtNf7GgAAG11R/qZJT7hwPq3Cc9u571toP5hq61tLs463ShS99S1cderoFBY9kU2HOglpSJ50gzEmmznVLsofYJlWiVV1rEwOrCdTpv2r6ibdumuL0YO+vf38XEVcFet/qXD+Ndb+4gAACStJREFU1Yyi9GMckPO4xqn3VMtqS6Y/XfVGo9GmJBLUTX0iE7L/qIa0tRv4o/VENdK7Evy2Tl5Muxjc2G/ScyJZVYruUxfBG+EzA238Lvp2ideAdjLSTjf7wWQnt9sj7Bg+VEFL/LaPst28/NAsAEB8M9PUcSEZ1wfe9yMAwOWbF6eefcB/r8jI7RDg+ubr96lVIn8CI4ywJXh2OxXPP6hnXv/eTVpTV59OKGkAAKhzkM3y/Grxwa/u0LU/eC3K/S0JTSfZ+mfE1ggAzLMu3e8AYwqP+GNWzXqskg9d08xv0iRhXrVeZkbzynm/fo9aWFhYWFhYWFhYFBGGPm/2eFNahMrfPg8A2PNbHnrgPM5UFn2ZVn7vmsWZzgEB0WjJvCUkFFqpJ9snnI6kn+lhBJ3MyDlP7GVM3tYY2YWd3WRO/N5sJs0dKaathzMRr7AOvUvIUmxaxdlo5Rj6c1T4hTpTljQV/cYVe9qbQTWqLtjroh91v1snOx40r75eYRiFYFZyKCXZU5ZIXnmuPGlEG71G2VplkHwNxcu8arxrZchK9jCtzRVSEOI5wNckUYCEoQ3u5W9Hd5q5DueZlgY6yFZ0tnIWrBG4nG6JRa5RaEYm7Soodj1KnVzLPPGdKf2IasLr/Nk8aaCzLwOgEX9aXVShrmY0iz/GpW+QwZgPN8O076Bxyt1t2N+ZzahW+cmOhgPiqzXEvE2roj5VPa1EE3y3ueSpqnH1yqpYcyff/xSJ6vXCZGlPXRqpkL9qa5B0OQixGD1UXkomERIFb6qX9d4vnVzEGdrnXN/pQAzXUPCHTjJ+x4aY1pVRssQ1snq4+UW25f3QOOR7J4JA+ywvOmdITZ7I72apeJCJiZ/43r3SD0rfZrazbZ+0h14YfCt5XvBdPK1tETWXp8wnu3rtBlraBz6X9nqd7KXVvics7OEwI/B5u1gOjz5JX/lz8Pyw7uNG7Y3Ms/9+tvUzw9S0emQMtUVI+jXiU781wTKIia9sjYJaLu8pPVbj9kwfGdyvXX9S6plhSKSyQWpdjcvzQiIofXIzy3mSnucd3GDFMq8WFhYWFhYWFhbjBkNnXvsZXZc+yJH4ige5vUI8z5mFNFnrmUzWNLiHOpiOmdyu2MCZnycifgtfXZ3j7p2ubTIDue30AY054/bxBqzNm/5RQx4LyWXNZJinT6PWo1tEjKpj1d8ybyRrO/N/ZZ8iwiiFXbMWPe54XWkYwAPCWKD8cb73vfNpuRqpErbUFcwkrWPN0PDlyU735GzL/NArDQAK63+xUIiXCgsmVsC9E/Rdsl14Q2LpHZMIQhonm4sJ6N2TtgQNlCo7wRn1ggnUvL44j36NnaS0HGF5lYGNlkt0psJkqSCov5FWwXOu4QzdA7IKSyOMEuPWfZsc0eOW9lJ3r6tAaj09x79HfnnvA29mvzKm9cNPOlMjZ3WLH0+3T9Xf//N4AEC8gqkNNrMcNnnFX7ArE5nFlLqXrmZI+9Aobfe304p82rLsm2jEPo3INRzfyxbDQ2Intd9nnUZm8Jq//RkAMM/P9rBw6YcBAFOQ6/uZAWHKlOEyElXMyfVNcLWlVMQoHQe47Dm+vJQR61476WcAgDl+CvfPXnMuAGC/L6V16EOFWtbng28q23h0Nr0L9Nay3XRMY511DH+7pgureDTLbX4ZxxZP/5vekubexTQn1qzq84zhMq4KbxftW/5yPq35P/u5t4/ofgrVny6uJ6Pb/gHqd0/8Apnd79Xx+By/tmd3dLGA6zcbH23kqnr4jy8MO43uaGOHH7UeALB6V7af15z1MAds12NhYWFhYWFhYTFusE9sRZ2lrwMAQq79Fa5JVPHZv48uppdTmzbdT+Y17OGsbGEJI1YENOa00CKVnvx8ULdQKSGhIP/WSY3wVJmVh/dzzbSG6JNtX0BjTU+/hVrp1nM4E+6ZyDmWSJpSbI8n0VfF5/bvWtEg1rB/XZX1jGJEKurVZvKecVeDUX2iutoThxuof5bM/MYPpMtDpZ0TlvAmj3pE4y3lE64knd3TTcaxdLP48/wbWZviqRVpvOOCywAAj95/l+zZBiAdYUhZg+5J4gM649oTSuj3cJJXNL2GLJZabh/7masAAOWrCqNBGwk8ZUyjRtFLeRuozO4hZ39x+CzWcKERylSPqxbDFvsOidX0V6oR5LQOH1bH9rBTzvNWUd+YaHX5zpA+39HffMuXg4DxiTebmHgQep2trvNE1UryvLafc5WxQtrsaCC+bTsAwCO/uno00CpSyouDpG00+77ESnpKefdfrwEAzMPwmcz+UHEf+7FX7uP2YpCRNUceBADYuUi0rwdzNa9sCtnnqZWsK46MJzbs5LLenItzRBRT1n2QYwg3a73jx/S1P/M1rn7pXZwe11JrHljm1cLCwsLCwsLCYtzAeunbF8jj5/WFFZx5vBiUqNRtMov1uzhomWJ4OzPmGipaEzZOtWqq/dQAXFFhRmpfcrGURcS4piDlpNavFfdy9qgeOX1TJgMA4jNplxiZEExfKvkuaSSz6jRszbpXKrd53kUxYPYlZD2VxVB2vFbeledQsunOKp5n9p8NAEiueAMAMP/x/Peu+YVrxx25zyvCWpGCW9fV++6jAQB7FrAbKzmBvpzrHifLmukFc9HDZDpKazn7L3uQfm7VY0p5gax+C4G4RDdbu2EhAGD9Dtb32qUursFlvbsv6vRnH/kgAGDCTK7oTHyl+NrRmx7y3j/6o08DAEItfAdl2yTCGpYBAJJdg2OwRgSXb8/QbqalSTxctIq23LzVllUHwLxPjQ7jOhCcZYwoOolVJGXhnzru2p6Drf3cbIht33V+2f0sA/c3x4kPLtSpZV4tLCwsLCwsLCzGDcxgLbsAwBizG8Dm0UvOmGOm4zh9HRTkwVugPIAhlIktj2zY8siGLY9s2PLIhi2PbNjyyIYtj754C5RJ3vIY0uDVwsLCwsLCwsLCYixhZQMWFhYWFhYWFhbjBnbwamFhYWFhYWFhMW5gB68WFhYWFhYWFhbjBnbwamFhYWFhYWFhMW5gB68WFhYWFhYWFhbjBnbwamFhYWFhYWFhMW5gB68WFhYWFhYWFhbjBnbwamFhYWFhYWFhMW5gB68WFhYWFhYWFhbjBv8PWi99aoQi8PIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x864 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = [], []\n",
    "for i in range(10):\n",
    "    X.append(x_train[i]) # 将第i个feature加到X中\n",
    "    y.append(y_train[i]) # 将第i个label加到y中\n",
    "show_fashion_mnist(X, get_fashion_mnist_labels(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读取小批量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T07:53:26.344986Z",
     "start_time": "2020-03-29T07:53:25.814085Z"
    }
   },
   "outputs": [],
   "source": [
    "bath_size = 256\n",
    "if sys.platform.startswith('win'):\n",
    "    num_workers = 0  # 0表示不用额外的进程来加速读取数据\n",
    "else:\n",
    "    num_workers = 4\n",
    "    \n",
    "train_iter = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(bath_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T07:53:30.696472Z",
     "start_time": "2020-03-29T07:53:30.191934Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.50 sec\n"
     ]
    }
   ],
   "source": [
    "start = time.time()\n",
    "for X, y in train_iter:\n",
    "    continue\n",
    "print('%.2f sec' % (time.time() - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_0nzlmif",
    "id": "7886893A27B44B979B7E4B93F870B3CE",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## softmax从零开始的实现"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_63x3cw1",
    "id": "24E441B54DB5429991AC9DEBB6014FEC",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 获取训练集数据和测试集数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:22.433316Z",
     "start_time": "2020-04-06T07:13:21.850875Z"
    },
    "graffitiCellId": "id_lkxl1n6",
    "id": "39EFD4C2466B4A649B15C7535800FAD6",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import fashion_mnist\n",
    "\n",
    "batch_size=256\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
    "x_train = tf.cast(x_train, tf.float32) / 255 #在进行矩阵相乘时需要float型，故强制类型转换为float型\n",
    "x_test = tf.cast(x_test,tf.float32) / 255 #在进行矩阵相乘时需要float型，故强制类型转换为float型\n",
    "train_iter = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size)\n",
    "test_iter = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_h6qpe01",
    "id": "7DC915895B5C46278F4C395BD49815A0",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 模型参数初始化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "跟线性回归中的例子一样，我们将使用向量表示每个样本。已知每个样本输入是高和宽均为28像素的图像。模型的输入向量的长度是 $28 \\times 28 = 784$：该向量的每个元素对应图像中每个像素。由于图像有10个类别，单层神经网络输出层的输出个数为10，因此softmax回归的权重和偏差参数分别为$784 \\times 10$和$1 \\times 10$的矩阵。`Variable`来标注需要记录梯度的向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:23.541710Z",
     "start_time": "2020-04-06T07:13:23.534730Z"
    },
    "graffitiCellId": "id_xcafpiw",
    "id": "FD4BCCEF1A044DF99791FB17D8134B6A",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "num_inputs = 784\n",
    "num_outputs = 10\n",
    "W = tf.Variable(tf.random.normal(shape=(num_inputs, num_outputs),mean=0, stddev=0.01, dtype=tf.float32))\n",
    "b = tf.Variable(tf.zeros(num_outputs, dtype=tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_d5s0vs0",
    "id": "3BBF52F8DB29448B9D44F5D707596788",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 对多维Tensor按维度操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T07:56:57.811824Z",
     "start_time": "2020-03-29T07:56:57.806837Z"
    },
    "graffitiCellId": "id_e13vki6",
    "id": "6BFBFFFB54E44D05B1FC16BE841BB815",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[5 7 9]], shape=(1, 3), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[ 6]\n",
      " [15]], shape=(2, 1), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "X = tf.constant([[1,2,3], [4,5,6]])\n",
    "print(tf.reduce_sum(X, axis=0, keepdims=True))\n",
    "print(tf.reduce_sum(X, axis=1, keepdims=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T07:57:09.061015Z",
     "start_time": "2020-03-29T07:57:09.056995Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([5 7 9], shape=(3,), dtype=int32)\n",
      "tf.Tensor([ 6 15], shape=(2,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# 不保留维度\n",
    "print(tf.reduce_sum(X, axis=0))\n",
    "print(tf.reduce_sum(X, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_fn39y6u",
    "id": "F15E94B777BA4E9F88CB18BD16697A45",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义softmax操作\n",
    "\n",
    "$$\n",
    " \\hat{y}_j = \\frac{ \\exp(o_j)}{\\sum_{i=1}^3 \\exp(o_i)} \n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:34.674759Z",
     "start_time": "2020-03-29T08:48:34.671693Z"
    },
    "graffitiCellId": "id_cr8iqvr",
    "id": "7F882A63347F46318C5F334F4F55F123",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def softmax(logits, axis=-1):\n",
    "    # axis=-1 表示最后一个维度,矩阵相当于 axis=1，即计算一行元素\n",
    "    logits_exp = tf.exp(logits)\n",
    "    return logits_exp / tf.reduce_sum(logits_exp, axis, keepdims=True) # 这里运用广播机制"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:35.649426Z",
     "start_time": "2020-03-29T08:48:35.643413Z"
    },
    "graffitiCellId": "id_vxj92pl",
    "id": "A77A982B8916433AAB521914CFD8467D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2, 5), dtype=float32, numpy=\n",
       " array([[0.13094425, 0.22753422, 0.1210086 , 0.08528198, 0.43523097],\n",
       "        [0.3038182 , 0.14748935, 0.10095752, 0.18262233, 0.26511264]],\n",
       "       dtype=float32)>,\n",
       " <tf.Tensor: shape=(2,), dtype=float32, numpy=array([1., 1.], dtype=float32)>)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = tf.random.normal(shape=(2, 5))\n",
    "X_prob = softmax(X)\n",
    "X_prob, tf.reduce_sum(X_prob, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_bkbmqh1",
    "id": "F59CC29AE6F5407A84064647281E9ECA",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### softmax回归模型\n",
    " \n",
    "$$\n",
    " \\begin{aligned} \\boldsymbol{o}^{(i)} &= \\boldsymbol{x}^{(i)} \\boldsymbol{W} + \\boldsymbol{b},\\\\ \\boldsymbol{\\hat{y}}^{(i)} &= \\text{softmax}(\\boldsymbol{o}^{(i)}). \\end{aligned} \n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:58.098617Z",
     "start_time": "2020-03-29T08:48:58.093632Z"
    },
    "graffitiCellId": "id_76a2fpk",
    "id": "0B3F76DCDD23439D849A2AE842D7DBE5",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    # 这里通过reshpe函数将每张原始图像改成长度为num_inputs的向量。X*W\n",
    "    logits = tf.matmul(tf.reshape(X, shape=(-1, W.shape[0])), W) + b\n",
    "    return softmax(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_9kc1jw8",
    "id": "5024E97053FF460A8CB5F2FCC75C0C27",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义损失函数\n",
    "\n",
    "$$\n",
    "H\\left(\\boldsymbol y^{(i)}, \\boldsymbol {\\hat y}^{(i)}\\right ) = -\\sum_{j=1}^q y_j^{(i)} \\log \\hat y_j^{(i)},\n",
    "$$\n",
    "  \n",
    "\n",
    "$$\n",
    "\\ell(\\boldsymbol{\\Theta}) = \\frac{1}{n} \\sum_{i=1}^n H\\left(\\boldsymbol y^{(i)}, \\boldsymbol {\\hat y}^{(i)}\\right ),\n",
    "$$\n",
    "  \n",
    "\n",
    "$$\n",
    "\\ell(\\boldsymbol{\\Theta}) = -(1/n) \\sum_{i=1}^n \\log \\hat y_{y^{(i)}}^{(i)}\n",
    "$$\n",
    "  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:02:07.986486Z",
     "start_time": "2020-03-29T08:02:07.927673Z"
    },
    "graffitiCellId": "id_fl3cdvq",
    "id": "0244A13F3B4F43EA9B8D2FC5800238C7",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[1., 0., 0.],\n",
       "       [0., 0., 1.]], dtype=float32)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2 个样本3个类别的预测概率\n",
    "y_hat = np.array([[0.1, 0.3, 0.6], [0.3, 0.2, 0.5]])\n",
    "# 2 个样本真实值\n",
    "y = np.array([0, 2], dtype='int32')\n",
    "tf.one_hot(y, depth=3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:02:36.956802Z",
     "start_time": "2020-03-29T08:02:36.838576Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float64, numpy=array([0.1, 0.5])>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tf.boolean_mask 在 y_hat 中保留后面的参数为 1 的位置的数据\n",
    "tf.boolean_mask(y_hat, tf.one_hot(y, depth=3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(tf.boolean_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:42.340554Z",
     "start_time": "2020-03-29T08:48:42.335540Z"
    },
    "graffitiCellId": "id_4ymtpzn",
    "id": "D288CFCAE8DF464287ECA08AD5168F2E",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def cross_entropy(y_hat, y):\n",
    "    y = tf.cast(tf.reshape(y, shape=[-1, 1]),dtype=tf.int32)\n",
    "    y = tf.one_hot(y, depth=y_hat.shape[-1])\n",
    "    y = tf.cast(tf.reshape(y, shape=[-1, y_hat.shape[-1]]),dtype=tf.int32)\n",
    "    # 注意这里没有对损失求和，所以后面训练时需要求和\n",
    "    return -tf.math.log(tf.boolean_mask(y_hat, y)+1e-8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_epbrym1",
    "id": "83DD3AF7D13F489D8683CB751F430CBF",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义准确率\n",
    "我们模型训练完了进行模型预测的时候，会用到我们这里定义的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:43.838856Z",
     "start_time": "2020-03-29T08:48:43.835865Z"
    },
    "graffitiCellId": "id_8h5q5ic",
    "id": "B82A0C9B1F984CF089B9FA8747BE1A3B",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def accuracy(y_hat, y):\n",
    "    return np.mean((tf.argmax(y_hat, axis=1) == y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:09:00.588897Z",
     "start_time": "2020-03-29T08:09:00.527062Z"
    },
    "graffitiCellId": "id_2szlvh2",
    "id": "9A6EF0B8618B400D8D252223B6B04E2D",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.5\n"
     ]
    }
   ],
   "source": [
    "print(accuracy(y_hat, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:45.291823Z",
     "start_time": "2020-03-29T08:48:45.287863Z"
    },
    "graffitiCellId": "id_gtpf1ob",
    "id": "963FEE28FD23446895193E95A4915F2A",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 描述,对于tensorflow2中，比较的双方必须类型都是int型，所以要将输出和标签都转为int型\n",
    "def evaluate_accuracy(data_iter, net):\n",
    "    acc_sum, n = 0.0, 0\n",
    "    for _, (X, y) in enumerate(data_iter):\n",
    "        y = tf.cast(y,dtype=tf.int64)\n",
    "        acc_sum += np.sum(tf.cast(tf.argmax(net(X), axis=1), dtype=tf.int64) == y)\n",
    "        n += y.shape[0]\n",
    "    return acc_sum / n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:12:46.382329Z",
     "start_time": "2020-03-29T08:12:45.985261Z"
    },
    "graffitiCellId": "id_90h5xb7",
    "id": "BDDE1D7C593D47B78465BA7E4F991B82",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0895\n"
     ]
    }
   ],
   "source": [
    "print(evaluate_accuracy(test_iter, net))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_1p8f5ji",
    "id": "A209780170A64CC29271427320284785",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:48:47.109976Z",
     "start_time": "2020-03-29T08:48:47.105987Z"
    }
   },
   "outputs": [],
   "source": [
    "def sgd(params, lr, batch_size, grads):\n",
    "    \"\"\"\n",
    "    Mini-batch stochastic gradient descent.\n",
    "    lr: 步长\n",
    "    \"\"\"\n",
    "    # 对每一个参数求梯度，并更新\n",
    "    for i, param in enumerate(params):\n",
    "        param.assign_sub(lr * grads[i] / batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:49:09.313689Z",
     "start_time": "2020-03-29T08:49:04.169357Z"
    },
    "graffitiCellId": "id_smslnb6",
    "id": "EFDCDFDDE4B44FA3866BEABC48F44D57",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 0.7846, train acc 0.749, test acc 0.794\n",
      "epoch 2, loss 0.5704, train acc 0.813, test acc 0.811\n",
      "epoch 3, loss 0.5254, train acc 0.825, test acc 0.820\n",
      "epoch 4, loss 0.5012, train acc 0.832, test acc 0.825\n",
      "epoch 5, loss 0.4854, train acc 0.836, test acc 0.828\n"
     ]
    }
   ],
   "source": [
    "# 迭代周期，学习率\n",
    "num_epochs, lr = 5, 0.1\n",
    "def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params=None, lr=None, trainer=None):\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0\n",
    "        for X, y in train_iter:\n",
    "            with tf.GradientTape() as tape:\n",
    "                y_hat = net(X)\n",
    "                l = tf.reduce_sum(loss(y_hat, y))\n",
    "            grads = tape.gradient(l, params)\n",
    "            if trainer is None:\n",
    "                # 如果没有传入优化器，则使用原先编写的小批量随机梯度下降\n",
    "                sgd(params, lr, batch_size, grads)\n",
    "            else:\n",
    "                # tf.keras.optimizers.SGD 直接使用是随机梯度下降 theta(t+1) = theta(t) - learning_rate * gradient\n",
    "                # 这里使用批量梯度下降，需要对梯度除以 batch_size, 对应原书代码的 trainer.step(batch_size)\n",
    "                trainer.apply_gradients(zip([grad / batch_size for grad in grads], params))  # “softmax回归的简洁实现”一节将用到\n",
    "                \n",
    "            y = tf.cast(y, dtype=tf.float32)\n",
    "            train_l_sum += l.numpy()\n",
    "            train_acc_sum += tf.reduce_sum(tf.cast(tf.argmax(y_hat, axis=1) == tf.cast(y, dtype=tf.int64), dtype=tf.int64)).numpy()\n",
    "            n += y.shape[0]\n",
    "        test_acc = evaluate_accuracy(test_iter, net)\n",
    "        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'% (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))\n",
    "\n",
    "trainer = tf.keras.optimizers.SGD(lr)\n",
    "train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, batch_size, [W, b], lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_eptwt05",
    "id": "D7885C25CD7444BF8212D2102E99692C",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 模型预测\n",
    "现在我们的模型训练完了，可以进行一下预测，我们的这个模型训练的到底准确不准确。\n",
    "现在就可以演示如何对图像进行分类了。给定一系列图像（第三行图像输出），我们比较一下它们的真实标签（第一行文本输出）和模型预测结果（第二行文本输出）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:22:56.408210Z",
     "start_time": "2020-03-29T08:22:56.125965Z"
    },
    "graffitiCellId": "id_yt8sd9t",
    "id": "1DA8927186304BEBA2B3DCC4A9E027DD",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAq8AAABwCAYAAAAwlplOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2dd3xlVbn3f8/p6cmkTZ9ML1QpQ0dEkCYIKKDo9cUKXhVRVJSLr1jhXr2K93oV5YoKyNBUXkSalAGGImVgBgaY3jOTSSa9nOSU9f7xPGuvnZyTTDKTTM4Znu/nk8852WvvffZ69lpr7/Wsp5AxBoqiKIqiKIqSDwTG+wIURVEURVEUZbjoy6uiKIqiKIqSN+jLq6IoiqIoipI36MuroiiKoiiKkjfoy6uiKIqiKIqSN+jLq6IoiqIoipI37LeXVyJaSkSfHWnZHs5ZR0SGiEL7foX5AxH9gYh+KN9PIaJt431NivJuh4iuJ6I7hihfRUSn7MdLUpR3JdoXx5aRvHuN1TuKal4HQW7MnPG+DiU7RLSJiE4b7+vIB1RWuYEx5iBjzNLByvf0wM1VtH3lPn6Fh3Lg9sV3E/ryqoyYXNd058r15cp1DEWuXGOuXMd4caDWP1fqlSvXoeQ+2lbygxG9vBLRt4hoPRF1ENFbRHSBr+wyIlpGRD8lohYi2khEZw1ynklEtJKIvj5I+aeJ6G05z6NENGMPl/ZpIqonoh1EdLXvPFEiuknK6uV71Ff+OSJaR0TNRPQAEU2W7c/ILiuIqJOILhmujEaCaCy+LbJsIaLfE1HMynLAvsPSBBPRQjHDaJWlkfNk+7FEtJOIgr59LyCilfI94Lu/u4noHiKaIGV2ieAzRLQFwJOjKogRQkS3A5gO4G9yf76Z7fqI6DyRQavIZKHvHP3kOcAUo4qIHpTjmonoWSIKSNlkIvozETVKG7/Sd47rieg+IrqDiNoBXLZfBDIEKquxgYiuIaLtMhauJqL3S1GEiG6T7auI6CjfMZ6GMkv9rwBwLYBL5D6t2P+1GjnavsYGIppGRH+Ruu0mol/KGH0dEW0mol3Szsp8x9xLPMa3EdEzRHSQbP88gI8D+Kbco7+NV73GAu2LmWSTCREtJqIXpC/tkDYV8R1jiOgKIlpL/D7yP0REUhYkfrdrIqINAM4Z8HufIn5n6yCiDUR0+ZhX0hgz7D8AFwGYDH7pvQRAF4BJUnYZgASAzwEIAvgCgHoAJOVLAXwWQB2ANQA+7zvvUgCfle/nA1gHYCGAEIDrADw/yPXUATAAlgAoAnAIgEYAp0n59wG8CKAGQDWA5wH8QMpOBdAE4AgAUQD/DeAZ37kNgDkjkc9I/wBsAvAmgGkAJgB4DsAPRZbLBuzrXQ+APwD4oXw/BcA2+R4W2V0LICJ17AAwX8rXAzjdd857AXxLvl8lspoq8vgNgCUD5HybyLlgLOUyAtnZ+5xxfQDmSfs8XeTyTZFNJNv9HSDTGwDcLMeFAZwEgMDt/lUA/1fkOwvABgBnyHHXg/vA+bLvuMtJZTUm8pwPYCuAyT6ZzpY6xQGcDR4DbwDw4iD3IaP+su2O8a6ftq9xl2cQwAoAPxcZxgCcCODTIrdZAIoB/AXA7b7jPg2gBDx+3wTg9WwyPZD+tC+OSCZHAjgW/F5VB+BtAFf5jjMAHgRQDp6QNgI4U8quAPAO3LvKU7J/SMrPkd8gAO8F0A3gCCk7BfKOMpp/I9K8GmPuNcbUG2PSxpi7AawFsNi3y2ZjzC3GmBSAPwKYBKDWV74I/KL6XWPMbwf5mcsB3GCMedsYkwTwYwCH09Da1+8ZY7qMMW8A+D2Aj8n2jwP4vjFmlzGmEcD3APyLr+xWY8xyY0wvgG8DOI6I6oYhitHkl8aYrcaYZgA/grv2veFY8KB2ozGmzxjzJLgx2nMusd+JqATcsZdI2eUA/s0Ys03kcT2Aj1D/JZTrRc49+3CNY4n/+i4B8HdjzD+MMQkAPwUPSscP4zwJcNudYYxJGGOeNdwLjwZQbYz5vsh3A4BbAHzUd+wLxpj7pY/kqpwAldW+kAK/ICwiorAxZpMxZr2ULTPGPCRj4O0ADhviPPla/+Gg7WvvWQxWEn1DZBg3xiwDP7N+ZozZYIzpBD+zPmrHaGPMrcaYDt/4fZhfM3uAon0xk6wyMca8aox50RiTNMZsAiuo3jvg2BuNMa3GmC3gF9TDZfvFAG7yvavc4D/IGPN3+Q1jjHkawGPgieaYMVKzgU8S0euidm4FcDCAKt8uO+0XY0y3fC32lX8cwHYA9w3xMzMA/ML3G83gt/kpQxyz1fd9M7jjQz43D6dMBoPde/idsWCwa98bJgPYaoxJDzinrdOdAC4kNp24EMByY4yVwQwAf/XJ/W1wJ/BPPvzXmov4r2/g/U1L+XDu70/AGo7HZAnkW7J9BoDJVkYip2uRXzKyqKz2EmPMOvBKxfUAdhHRXSQmR/CNgWDtQ4wGt6HLy/oPE21fe880sCIoOWB7tudZCECtLOveSGz21Q7WLAL9n88HHNoXMxlMJkQ0T0xwdkob+TEy28dAmdn3t8nIfFfxIKKziOhFMe1pBSvGxrTtDfvlVTSftwD4EoBKY0w5eMmbRvB714OX6u8kn+3lALYCuNwYU+77KzDGPD/Eeaf5vk8HmytAPmcMp4yIigBUgl+u9yfZrr0LQKHdSEQTh3muegDTSGzCfOfcDgDGmLfAje4sAJeCX2YtWwGcNUDuMWOMXx5mmNexP8h2Lf5tA+8vgWVt69MNn4wBeDIW7cXVxphZAM4F8DWxo9oKYOMAGZUYY87ew3WNNyqrUcYYc6cx5kSw3AyAf9+b0+zh/3xB29foshXA9CwvWtmeZ0kADeDx/EMATgNQBl4WBtzzOV9lsUe0L2YyiEx+DV76n2uMKQVP9ob7/rYDme8qANi3CMCfwSsqtfJu+NAIzr1XjETzWgQWQiPABrpgzetISIDtZosA3D7gJctyM4Bv+4zNy4jooj2c9ztEVCjHfArA3bJ9CYDriKiaiKrA9lE2/MWdAD5FRIeL8H8M4J+iTgd4QJg1wvrtDV8koqnEzlHXyrWvAHCQXFsM/NI/HP4JfvH9JhGFiePYnQvgLt8+dwK4EsDJYJtXy80AfmTNM0RmH9r7ao05e7o/9wA4RwzVwwCuBtALtnsGgNcBXCoaizPhWz4hog8S0Rx5yLaDNdApAC8BaBdj+AI59mAiOnr0qzeqqKxGESKaT0SnyrgRB9ADrvO+0gCgbpBxMZfR9jW6vAR+WbiRiIqInXhPAD/PvkpEM4moGPzMuls0tCVgme4GTwR+POCc++t5tl/RvpjJEDIpAfehTiJaAPZLGi73ALhS3lUqAHzLVxYBmyk0AkgSO+p/YBSqMiTDvjGitftPAC+Ab+whYAejEWGM6QMvWdcAuHVg4zDG/BU8S7hLVNtvgjWFQ/E0eGnpCQA/NcY8Jtt/COAVACsBvAFguWyDMeYJAN8Bzxh2gI2N/fZS1wP4oyxJXTzSeo6AO8H2IRvk74fGmDVgZ7PHwXbFywY/3CGyPQ8sryYAvwLwSWPMO77dloANqJ80xjT5tv8CwAPg5bkOsPPWMXtfrTHnBvDEpBXARwYWGmNWA/gE2BGvCfwSf67ICAC+IttaweYs9/sOnwuWfSe4vf/KGLNUbKfOBdsBbZTz/i9Y05HLqKxGlyiAG8F12gkey64dhfPayeRuIlo+CufbX2j7GkV8dZsDYAuAbWC74VvBtpvPgOscB/BlOew28KradgBvgcdvP78D20C2EtH9OHDQvpjJYDL5OlhD3wFeRb97sBNk4RYAj4IVa8vBzoIAeHUErBC7B0CL/MYD+1qJPWEjASjjABFtAkdZeHy8r0VRFEVRFCUfyDuVuKIoiqIoivLuRV9eFUVRFEVRlLxBzQYURVEURVGUvEE1r4qiKIqiKEresN9eXonzWX92pGV7OGcdcT7ewQIPH5BQ/9zfpxDRtvG+JkV5t0OcI/2OIcpXSfg6RVHGEO2LY8tI3r3G6h1FNa+DIDdmznhfh5IdItpERKeN93XkAyqr3MAYc5AxZulg5Xt64OYq2r5yH7/CQzlw++K7CX15VUZMrmu6c+X6cuU6hiJXrjFXrmO8OFDrnyv1ypXrUHIfbSv5wYheXonoW8S5kzuI6C0iusBXdhkRLSOinxJRCxFtlEwL2c4ziYhWEtHXByn/NBG9Led5lCTr0xB8mojqiWgHEV3tO0+UiG6Ssnr5HvWVf46I1hHn432AJCcyET0ju6wgok4iumS4MhoJorH4tsiyhYh+L9lULiOiZQP2HZYmmIgWihlGqyyNnCfbjyXOaRz07XsBEa2U7wHf/d1NRPcQZ/3yLxF8hoi2AHhyVAUxQojodnB6ur/J/flmtusjovNEBq0ik4W+c/ST5wBTjCriHNCt0jaeJUmmQZwj+s9E1Cht/ErfOa4novuI6A7iBBuX7ReBDIHKamwgzuy0XcbC1cQpSwEgQkS3yfZVRHSU7xhPQ5ml/leAA4lfIvdpxf6v1cjR9jU2ENE0IvqL1G03Ef1SxujriGgzEe2SdlbmO+Ze4jG+jYieIZel8vPg5A/flHv0t/Gq11igfTGTbDIhosVE9IL0pR3SpiK+YwwRXUFEa4nfR/6HiEjKgsTvdk1EtAHAOQN+71PE72wdRLSBiC4f80oaY4b9B07tOhn80nsJOBXpJCm7DJz+9XMAguDUY/VwEQ2WAvgsOOfyGgCf9513KThYPwCcD86WtRBACMB1AJ4f5HrqwClrl4BTzh4CTlF2mpR/H5xppAZANTgd4Q+k7FRwBoojwBkp/hvAM75zGwBzRiKfkf4B2ATOIDYNwARwxrIfiiyXDdjXux4AfwBn4gI4W9Y2+R4W2V0LTtl2KjibxnwpXw/gdN857wXwLfl+lchqqsjjNwCWDJDzbSLngrGUywhkZ+9zxvUBmCft83SRyzdFNpFs93eATG8Ap8sNy99J4DzNAQCvgtMMR8DpFjcAOEOOux7cB86XfcddTiqrMZHnfHD++ck+mc6WOsUBnA0eA28A8OIg9yGj/rLtjvGun7avcZdnEJzJ6OciwxiAEwF8WuQ2C0AxOMvR7b7jPg1OARoFcBOA17PJ9ED60744IpkcCeBY8HtVHYC3AVzlO84AeBBAOXhC2gjgTCm7AsA7cO8qT8n+ISk/R36DwOmduwEcIWWnQN5RRvNvRJpXY8y9xph6Y0zaGHM3OHXpYt8um40xtxhOb/dHAJMA1PrKF4FfVL9rjPntID9zOYAbjDFvG87Z/GMAh9PQ2tfvGWO6jDFvAPg9gI/J9o8D+L4xZpcxphHA9wD8i6/sVmPMcmNML4BvAziOiOqGIYrR5JfGmK3GmGYAP4K79r3hWPCgdqMxps8Y8yS4MdpzLrHfiagE3LGXSNnlAP7NGLNN5HE9gI9Q/yWU60XOPftwjWOJ//ouAfB3Y8w/jDEJAD8FD0rHD+M8CXDbnWGMSRhjnjXcC48GUG2M+b7IdwM4bZ4/rfALxpj7pY/kqpwAldW+kAK/ICwiorAxZpMxZr2ULTPGPCRj4O0ADhviPPla/+Gg7WvvWQxWEn1DZBg3xiwDP7N+ZozZYIzpBD+zPmrHaGPMrcaYDt/4fZhfM3uAon0xk6wyMca8aox50RiTNMZsAiuo3jvg2BuNMa3GmC3gF9TDZfvFAG7yvavc4D/IGPN3+Q1jjHkanPL+pDGs44jNBj5JRK+L2rkVwMEAqny77LRfjDHd8rXYV/5xcO7l+4b4mRkAfuH7jWbw2/yUIY7Z6vu+GdzxIZ+bh1Mmg8HuPfzOWDDYte8NkwFsNcakB5zT1ulOABcSm05cCGC5McbKYAaAv/rk/ja4E/gnH/5rzUX81zfw/qalfDj39ydgDcdjsgTyLdk+A8BkKyOR07XILxlZVFZ7iTFmHXil4noAu4joLhKTI/jGQLD2IUaD29DlZf2HibavvWcaWBGUHLA92/MsBKBWlnVvJDb7agdrFoH+z+cDDu2LmQwmEyKaJyY4O6WN/BiZ7WOgzOz722Rkvqt4ENFZRPSimPa0ghVjY9r2hv3yKprPWwB8CUClMaYcvORNI/i968FL9XeSz/ZyAFsBXG6MKff9FRhjnh/ivNN836eDzRUgnzOGU0ZERQAqwS/X+5Ns194FoNBuJKKJwzxXPYBpJDZhvnNuBwBjzFvgRncWgEvBL7OWrQDOGiD3mDHGL49cymiR7Vr82wbeXwLL2tanGz4ZA/BkLNqLq40xswCcC+BrYke1FcDGATIqMcacvYfrGm9UVqOMMeZOY8yJYLkZAP++N6fZw//5grav0WUrgOlZXrSyPc+SABrA4/mHAJwGoAy8LAy453O+ymKPaF/MZBCZ/Bq89D/XGFMKnuwN9/1tBzLfVQCwbxGAP4NXVGrl3fChEZx7rxiJ5rUILIRGgA10wZrXkZAA280WAbh9wEuW5WYA3/YZm5cR0UV7OO93iKhQjvkUgLtl+xIA1xFRNRFVge2jbPiLOwF8iogOF+H/GMA/RZ0O8IAwa4T12xu+SERTiZ2jrpVrXwHgILm2GPilfzj8E/zi+00iChPHsTsXwF2+fe4EcCWAk8E2r5abAfzImmeIzD6099Uac/Z0f+4BcI4YqocBXA2gF2z3DACvA7hUNBZnwrd8QkQfJKI58pBtB2ugUwBeAtAuxvAFcuzBRHT06FdvVFFZjSJENJ+ITpVxIw6gB1znfaUBQN0g42Iuo+1rdHkJ/LJwIxEVETvxngB+nn2ViGYSUTH4mXW3aGhLwDLdDZ4I/HjAOffX82y/on0xkyFkUgLuQ51EtADslzRc7gFwpbyrVAD4lq8sAjZTaASQJHbU/8AoVGVIhn1jRGv3nwBeAN/YQ8AORiPCGNMHXrKuAXDrwMZhjPkreJZwl6i23wRrCofiafDS0hMAfmqMeUy2/xDAKwBWAngDwHLZBmPMEwC+A54x7AAbG/vtpa4H8EdZkrp4pPUcAXeC7UM2yN8PjTFrwM5mj4PtipcNfrhDZHseWF5NAH4F4JPGmHd8uy0BG1A/aYxp8m3/BYAHwMtzHWDnrWP2vlpjzg3giUkrgI8MLDTGrAbwCbAjXhP4Jf5ckREAfEW2tYLNWe73HT4XLPtOcHv/lTFmqdhOnQu2A9oo5/1fsKYjl1FZjS5RADeC67QTPJZdOwrntZPJ3US0fBTOt7/Q9jWK+Oo2B8AWANvAdsO3gm03nwHXOQ7gy3LYbeBVte0A3gKP335+B7aBbCWi+3HgoH0xk8Fk8nWwhr4DvIp+92AnyMItAB4FK9aWg50FAfDqCFghdg+AFvmNB/a1EnvCRgJQxgEi2gSOsvD4eF+LoiiKoihKPpB3KnFFURRFURTl3Yu+vCqKoiiKoih5g5oNKIqiKIqiKHmDal4VRVEURVGUvEFfXhVFURRFUZS8YbBsE1mJUNTEUDRW15LTxNGFPtM7oqC7YykvCnGOh1RJDAAQaOka3oElEgs8JUm4uuOjfWkARi6vsZRVuoLPS9UJAEBfT9gVhlgO1MfzOOOfzgXFpEY+IhFOeENr+zCa5ErbogjLJV4T4f8lWiKls+ybLZIi9f8MsLgRbO32dhkNM6VckddATCn3LRN0l2YGyKTf/lZOSZZJwCen0STX5BVZwJ0sLUJJpV2nCwa4sXX3cRsMBFx7iYW4QSXSPPaRdEx/xcyaxD5fX67Jy0JhflybhEu8RQU8/qfDIkNf9yLb1zrHNttpToz1g6RiSNS430kXyVgv+5qku+SAXH64YZjP0b0kV9tWrjKUvEb08hpDEY6h94/OVeUZ/zRPjPiYfZVXoIgb7PrvHOpt+8w5HFXr4ALO1HZMdDcAoD7lGvehkdig52xKcedsSPFgFzclAIArV7sQt+k/1gAASpcMDBU4fEYqr1FrWwFf4rY0v2HN/gfL41dTBq/P+kQnAGBSMOJtKwzw9x1JKQuxrI65xsV2Lr/9hX2+5PFoW9nY9g1ONb/sCz8FAKzo48yAT3Uu9Pb5aNnLAIBHuxYBAP6y7T1e2aXTXgIANCVZTr97gVNbl61yk4Xa/xoqUd7wGHV50RDPkgEv28HKCd73pnPnAwB2H877fOS93L4e3uzkZeQNtaKQXyDmljV6ZatbuZ+Vx7gsEuCXkncem+vtU/dn3j/19trBr3EP5Er7ClZVAgB+8MjDAIDnu+dm7DM53AIACBPLojFZ6pXFDbejHX3lAIAJIR7LXmlzSad2n9Cyz9c5HvKikLyYJn0ZYaVd/nkrjzHFAR7H7HgEADXBAgDAO4leAEBZwM0qO2RScPYj3wYAzLvipb2+vqEYt7E+G9KV0yceDgD4xz1/8Iq+tJ3DlttJU2fSjfU3T+Pn6oV1ZwAYcB+8c9u33r2fgOdKX8wXhpKXmg0oiqIoiqIoeYO+vCqKoiiKoih5w4jMBpT9w5qbFwMAHjrzJgDArPBSr6whxctDO1NRAMCKPl5Wmxh0S0nbZFkpIsscrT6bxXpZ0rXLchMCbPP64EF3ePtEf8LN4itXvQ8AsOWYsbUDGlXSmcaY36rlJaGVfVyvl3vqvLJpYTa7iAV4ifzVXpddsjvNMg6gCgDwyVLOpts63527fJQuOxeIV3FDubdzDgCgNx3O2OfJbq58WIxe/cvgc6M7AQDr47IcPrEDAJBcOwE5jV0GHGJZcPs1bFLRNdtnUxnk70Vrefnx/gePAwBED2n1dunt5TbXKeYoz26c7ZUlOnhbfQ+bugSquG+nprnfaPs5y7m9h800pn3D2S+m1m0cXv1yBCpim+CULNsWBri+balCb59W+f5W92QAQDTglm9nx3YBANJiivFoA8ukrdeZSZVh380GxoNsy9SNVxwLADh8CX9WHcR97bnD7vH2mfO3KwAARx+8HgBwzyy3zHrxBl5qXvjfbQCAdJjbm0n4bPZHYSk8F6CjDwEAdE9hM4qCeu4np172WW+f4n/bBgCYXshtpCle7JUd99OrAACl53F/23E8y2Xuda97+6TjY+MfouwdqnlVFEVRFEVR8gbVvOYIVrMDABvP+xUA4Jk4ayG2+rzj0+DZYgCsJSsVzWmjz2GrUZSPVsOR8rnQF4m2w9s3zb+xORn1tlnHiF9OXQoAOO+JC9wB7982kmrlBNNDLLPGXtY4WA0hAETAwtqdZvnFyGm9KsOswd6dcjN0AOibMrrRBnIFM4Hr/moHO8BcXMkOHm/Ep3n7zI2w7Db0sXZ1XlGDVxYUV9+6GGuo02YBAKB6ZY7Lawjt07ZruV/2VnBZwRbXF61S0Havgkbpb89VePvMOXsTAGBDIzsrJRM+h0L5uYpVfFz36XzCUIPTJDa01/JvTePVj403uLY4/aJh1C2H2HQpt6PDxE/m7+0sk8KAax92Rag1weNSyOeAVCOOqIVB3n96MWvQaivbvX1eOJVXrUJPvjrq17+/se2rcCe3j7JvrAMAfGjpOd4+/3oia1ovKV0hW1z7aP/yRACAWbVq8B/JQ41r8tQjAQAbz3evL+Fa1rRGX+Jt8XKWQ8Fu137eWD8VALCtilfXunrcM2/6a3x8Ksr9MyyrImt/4BxSpzzD5yp+bTtfx7bto1IfZe9QzauiKIqiKIqSN6jmNUf438v/2/u+PsGzwIThGWIs4LSBJw+IgrWqj7UQfWmn0bG2mtNCbHtXHXRGr6/3spVmRGwWrZZ1gs9m1mrQlsXZfuhXc+7yyq6cegmA/Jh1huqmyze2W+pIs/BSvsiQVg5W49pl3Gw8Ybh7pEW1ZsNpTajqGLuLHkfCW1nbkFzAbcnW32/7ujXB2jJrpxj1tc2nO1nTeljhFgBAgLgdxZY728xs4WHHHbIxMvnqgvPneEXdU1n9VbyJZZHIEm4xJOFZe2q4vqUbXNk7W1j7dWgd95fNrU4rG1/PtsAtJ4gt3U6WadA1QaQLuO+m49JPa9q8soYrWSvshR/LcfvFCy55FgDwdA/Xc0Ura8LeU77V28eOXadXsLZwZ9LZoNt22JRgrVpPiv8/sXiNt89fj+HwbFOfHP3r32ey3J9AjMek+ClssxmvcON4KiohnWZyuzTHHQYASJyywtvnQ5tXAgCe7J4FAFiyYLLvB1mGdNTBfL4Yt+HIJmenDontnNwk9yCLz0CuYH1BJj3F/TW62+neYmu4TVS9wc/O1lkSVuxSt9JYtII7b+86HsPI1882nsf9bNpj3N8r3uH/dy12+2x7P/9e6Fh+rsy8z3k8mNeG0G4rY4JqXhVFURRFUZS8QTWvOcL8sJshNoui1Hp0+7Wts5/4FABg1m/5/wfv4i/bfXaxZxbyuTYm+Pj7O+d5ZScUsFdqq2g4ThHNzmPdzuO3McURDKx9Y23QNZOeRZP42vJA89p21KR+/7eL5nViyGmvrObZfkZ8qaOsXbHVfO8Wmc2u2O1+Y7QvehwRRSmeWsPtpSHOkSmsBhUALqxdDsAlyQj7dKm3tZ3Q73xtm1lrNqmvfmwueLQYoG3qOKjS+26bg13YCPsCb6TEdtNIWaiLNWW9vhAUNY/xTkddsxkA0Npb4JV1hySzlnxGGvhEySKf5rSE214wbDNPuX7ecSivutTaDTmqcbVcMYGD7X+n/iwAQGWUhVkWcpnF7Ji3NcFa6bKgi65gbWPXdbO99bZOFnRftdNW9kzKXc1htvuTPIYTWjQczfe1fI1bJatYw/WtXs5toHMGt51o2dHePp+5igPvlyxjdX9oqgu8b4qkrbWzDOM1LNPEIU47G+yVMa6PfyPXVtTMCYd73685+e8AgL9eUQ0AKD3yIK+M5FmXXvkOAKD6HW4jvRVuFWXSz3iFInAorxClipzqtfFI1spGHn0FAFBQys/A4q0zvX3CG/h52H58HQBg3dedrGd/fC8qp+wTqnlVFEVRFEVR8gZ9eVUURVEURVHyBjUbyBEqgr5l+zQvpwVhl5DcHGP+1zhUVaqRje6jxMtNE0POieiTmz8AAGg4zoWQsSTe4iW2L4qTxNmHnAoAWHuNi7y/9hO/BgC8JJYMYXLLcvUn8u/NeGzYVRs3mg5lubWledmsMcnOM1NCLoh8ZUByzofYGWtFn1syTovcrflApYd5+cwAACAASURBVIQZa+xx4WgicCYE+Q6ledk7tJ2X0za+I0tmvtXOxz7IMji4hE0Brprwhld2TSPL97k3OGd94Q5pNwU+u5f2zDaZa+xe5Np7sIcrn7YrjL6oXyFZ0bahsmxUp0SJ24d28Ofvnj+Z/086Z8FoSr5vEPnIv4ka5wQXFJOCaMyXHEE467A3AQDr91ij8SM0q877HqPnAABvN7Ohw8IJHGYtYZy8rSPgB4u5XdlQfgCwqY+ThRSFuA1Gg+xc05gs9fYJ9OaXPiZeyUvPZet5jOkrdu2jsEG2lfM+BY3cBqI7nXPtruPZFCD9Pk5+Uf6KCwPYN5nl0lfGj/loEzfecIPrg5Ti3/BMDHKMhsXu/j/wETZL2nE1t4Pqs1zYxu3NbKJk1nKiEMzhZ+jti3/h7fN/Sr4CAIhPZzmUVzk5njX9NQDA07vYCbJlLrejnpmuw4ea2SmubDX/X/nIAO9pZb+SXz1dURRFURRFeVdzwGpeKeSqZlKiEsliMB8o5JldupudBug9bAS+v0Jf2FApfhKiyrGpWwE3++xdwjPk0Gn9jzk04gtsLhrXtb/gtILhDjebv/9yPudd1TKbn8f/z17i04h9gj8ionGMG+dEED4kf1yUit7DWtGEXP+UMAc17zLO0H5+mOv/3QbWjF1Xs8wre0MCpcclScEkSfG5ud5pZ+di85hc+3hgw8PsOIU/y99hjVio1/WbF9exNvatV9nR5LPfWO6Vtdazpqdgu2h6Wvg40+E0HPlAzxSXqjPUzjJIRa0MXF8q2MXbkoW8LS1Dji96GJoXcVn5mxJ2zEXKQuEOPr57Eu/TV8Zyr6h2qygtDSzT4+ZwOKgXttd5Zavb2CklYtOm5mD6yt7pLjXwtmT/x01AVPq7+pzm9PAiDrP23W3nAgC+NNmlO50ebgYAbAxxvYMBcWJLO8ebYP8cLDlJsMI1gt4SHuuLt7OGr+E8J6Oq5VyZZEEh/FCb60+pCI9FCas47XONLx20obn4I9TJv0EdzuvQlLCjUrokN7WI6ZPdKlnnRnbQq1nO7by1bYpXNqGTK1n+Kmvzu+eyXD6660vePnMe52d8yzwWVjrs7sNdB7HGddYullFXLcsj2egcJH0LBHw9U91YUDWVryXXHN4OZFTzqiiKoiiKouQN+aF5tcGdbTBxX2ib4Fy2Q9l1CttR1dz7FgAg1To8DaHVuFo2XMxagJmv7fXVjgiaPUO+vehts5rX2mCmndtxVRzw/WX0nwYe9d0veN8rwSFp5v2BNTiBLp9GJsTHBZ7lClqbNNM2PFvE909nDdDbw9p7fPnwDA7m3ZHmWXmfTJ0XhZzm4ske1uK8eaRoveqdliMi4VdsysrCAGteqcVpbg8kEqJBDHRz++ueyP9HWzL3tVrGioBPYyPd1Gq/bHpLKvJF9h/Q33IJl9TCkSridkHlYi+4ytkGWg3rQI0MuYUKBHv7h89KRXyrPyKvdFg01BNZcHFfOCyKchucV8S2jC+gzl2v/FDfCbxaFHoi91KiNi9wWtEuCUfX3i1tRpSyaeM0WKcW8ErGbSdyKtlnVi7wyj5TzmGM/pbk43uSNsyde4xZeec01U4bnRLxxLby8ypU7fpKfBKPReEO7kiUFPvULteHIu3cdoIJaVfks6nexRrWgt0ytgfk+Rl0DdZEeSxLh3lbsNCNfwOfjeNBVbHTErdM4040cTsbmxfXu3okC6RujbzaFqliw3NKuP4afH0tAKCofBEAINTt3iPa5/CNsPbBsRaWdSju5Fn7CPuJbPoEjxPdM93zOTVRtLiqed1vqOZVURRFURRFyRv05VVRFEVRFEXJG/LDbMCSJe/yztPYXKDlKMlNP4mX0KZ///lhnTI0g5entn+IP8P7OW19fFLxoGUlAb49nWm37P+BUg4h83LgyH772iUNALDuJpfd9RAA4KMlbt339V5emvza5V8EAPzhf28CANyw633ePluSvKxuQ2R1++R+Uok1G5g1ZL1ygfkxjlPULeu6CVlenB5yMj/7lQsAAFOQ6aAXE3OBeNqaCfB9SEfSGfseCIS7JCxUoSyVp8V8YJJbOrPZoKzTRBpOFlTA8kqHWc7ecnpvHnjRAOhayGNJsNMtR6ZjXL+CQgmZY9wyZO8EcdSSVf6UL1e6xYbRspm6fMnK0FPd36EmHGH5hUOuv6UlRNbmHg4PVBBxS5W9KZZzx2xun1XOtylnaJvnKrw1wU40pYXcdnpSLLhjylx4p5d7a/od/4c3jvO+f/sUNgmzobVKItyu0sbpYAKZllY5R7LSjT9BG4mpiZ3REm0+k4KYmO00yYgekPaSSvn24c+whHQzvqV+KuRCGwbLOmolp1V7+xgxFwg18ZhPle73c8FsYFe7k5UR05tN5/K2WJMbl5LSLYu2sxle2xw2f4hOdw90s4ifWbsXcbuLNvvMTeq4/iW/ZhOL3jpuq7uO8DlCn8nvCIHF7EQW7HHmY6kYn0u1gfsPlbWiKIqiKIqSN+SF5pVCPFMyCZ6mJk5zWse2+TzjDEtIi97ZPKvvfazO22dnKxtvF8b4+JZtZV5ZuIJn72UlTXy+ele2P+iYlun8488lDwD1vpn2yTIR/JFoQ8+YzLmf6SiXUH3zf7Lx+O8l78DvMcMru+AtTm6weyH/7mePvwQAsPqr07x9/utjLwMAVvZJSJK0m+OcUchheX6bB5rX42McSL9ektCnkOnMUXJvSb//W1JO23CIhB97NW6dGMTxoSCH86fvA+Fu0aLKR7gzMzxUvJOHjODSlzOPj7GGKCgLBdZxK50nmteO6VI3/+UG+reZ7qnu3hduFScX0Z6lRWHrUwQiLM3JagS7proyE2D5BiRxQV8f/34o5LTZc2p4XNrVy9qm3qTTCveKuqlH0rdXDVm78aFolnOcXR2fBAAoCLMw4qI5Pr1wjbfP+5/iQPJzwc5n03/vhBl8H3+PBlwoMwDoTrsxlPKgayaLfBq/PuljCXHKirkKdFdJOMMdXG+SUI8mW8jHpI2H5XukB+W4Hm7QJsFyD3Q79TQlevqdx+SAttWPed09j+t+zdkBWk7nJCjtM13f7K3g+ndNLZD/uaysyNWvV7TKCVHmmqA7fsoEbqfxWayVbp7PyyhJX+6GuERIDL7Gz9oin094eBeH6MqD5nfAoJpXRVEURVEUJW/IXc1rwBfOQzSuwXKeha35iCsj0ZLYIOIFxbyBfNrLgGg47LY583d4ZRvqWV/R0iYhSkKZs9qxJF6dqQ20obKiYnNaSE7TYO1R1/7yGACAkev93PFPe/s8UsUz1G8sfw8AoC7W5JVdUc6hPBZceTMA4N9v4UQGkw/O1ADHRI2R8KmSigO5Gcw6G5PEtnVzkutRFMjUAJb/v5UAPGUjvrLtTK/sF1MfAQDEBhjSBZvDOBChlPSThFUhov8ngEBX/7hQjSkn04i12eyyGkXRFCX6a8pyFWvbS2lX4XAHt31ra9pd4tNaiarVS04gapeUTyVgw2ZZe1gT9J27k3fsKxdb47icL+bGhIZOXhmYXcF9uLPdqYLseFZzyK5h13F/U+0LddTYx3UxEhorJuldS3za7fk/4/1tfww/7sJ/JYwNXceffSmWV5tPPZYPmtdUge/5ZdtMJ9e7qtqFLOyuYC1gaDeP+YlaDuMY9CW2sY8G7xHR4wuLaFO/igY21cRhpFrPnuftUvmitB2rzaXcCDVmTuAVxVPPc/d/4+/5WW0fR2nf20ukna873MECjbTxToVh118TUWujLnXtcXWtb+F3ixnNLL9AUsIipt0+wT7+PuPvrHLtnurCmtWfyfbytWtyL1lzaMpkAEDPwknetmQxt8F2WW2qWsn13vQ5Nz5N/ROXFW7mNkm7XcKI5M6G/j9i39ey+CV5+NtWltWDkaKaV0VRFEVRFCVv2D+aV/vGbd+2fVpV2NSjUmbTuppkprZm/dUcXDjqUzQEJYhw93QJJB/lmda2Rpf6LRDk30iL7WZzt5upp/v4WqIlrEGyHr9WywsMP+HB3tBTm+m5br1prbd/Ebk5xuoEq3A2XPibfsesSTgNx3Nxrt+Xq57NOPczcdZGLo7yTOvhdZlRGVJyT2Ki2UlkmSQNdZ9ylRJJt9ud7vO2DfSofWW7C1QfnSY2kOh/j8LtB+acL9YgKZKNaMgC/b3pgcwg8BuSzhvYagKtzWvxNmsMmgfqMACJItEY+xTtUVE2HDeRk4M8+9DRXplXdTusyXEp3yKG1bhaLRElnfysjayXela0PH29TuDJtXwvqk7b3G8fAEiLirdM+vL+XTMaHnFfStidca6LTUpQE2NP8Kd7nEYovfKdQc/1Wh/3Q+sTsL2Nx+gFZU4LlMqDhaFU1N3DcJdE9pC+NrHYecfv3MmaxmSVNdK07cSXpEcGZxvZw28PG5BUsSbc/zHfdb7T7la8xc+KwEb2D6DC/qlox4tECfeBE0udPfQ/rjgCAHD0qZwiZ8t/OA1yyUp+KUhu4qg7sRh3vC3iEwIA0x5hLe7k5KG8z0439jc2s1bbvMYJfmq3SUSGGhd94fglnPTmj/N4tfK8+a94Zfe/dRgfN6JajoCBGvEBmstsz+OuD/PqbOhy7h/nTF7qlU0O88DWmuL7/bedLJOLK1zUokdmnAAAaL+C/7/5kAe9smu+8K8AgMgj4vuQZYwPyAqBl7Z6FLSt/c4/qmdTFEVRFEVRlDFEX14VRVEURVGUvGH0zQYGmggM/A5kVTEPtQy961+PBwD01fAySPlKt6xml+NCpbxE2dzCRtTGl3/eVEpecgn+HQ5mUXGLU1dxAZsPJA5zoaACT7+Wsf9oka7qG7SsLc1hPj6+7iPetptn3wMAeKSb43bEJV94ecDNQwrFMWlDojTjnHbpfFmc5VQZZHOD9QkXuHqNhLS5roqX8F7PEuqIDuJwJWbF24Nef65gQ2SVinffHR0zB903Xu+M8K3ZRupdMscLbGJHxnREZCDmKslC13/TA0aMInLtt7ubl+rK2sVRJJAbzh/DxSYkCMbd/SYvPjzLoOoNF3pn+3t5yTXsLHb4GJ+VSV8ZHxdpFWcwv/ObDEOBhJhnZBFXGadjx8RzxHTJfwIxeZpRzAHuNw1Sr/GkscWFoouF+o/t06N83de8/GFv22wMPtY+3bUAgDOr6mzidvpOmVusNXnQVZM+s4HYbrE1kb72iUkveGU3xesAAIEelluqSJ57oczHthf2yRdW0Tpqoa1/5p2/HelMzj4f+xL/vDh6UVnmM2M8KHhlAwDgP37+UW/bnPt52xtNbD7Ycb7vuXQqt4HCenZOsn3wxAtce3pjIzswt81iufSc4Uye3rv4TQDAmhZeau+YJoltfDmEjpHGVfEY9/unnj3GK5v1Rv+QY2MBBX1O7Kn+7zD2vSn1viO8bQX/yqYg69ZNBAAs+esHvLKaF9l0JNjE48qOD7O53D2L3XvAUZ9gx++Gbu7Dy7rme2Xf/Z/fAQDua2Yzqpd+yb9b8UfXfj1zgTEiD7q6oiiKoiiKojCjr3nNZpQrDlp25mCSiYz9B2pcd1x9vPe9Yw6XxbZLIgJnQw2x/UesgDVAnTtkqlTsm4HKLKyzhzVDBVGfttNTFPdXe2w+01n+z3waY0ZxWeaMbUaItz3cxYkDGu5ySQamf5frV5/s72gU9sWICXoeJJkaU6uFtGGjJgRYFl0h55R27WMfAwBcd+ngzhPxiaz1iKwYdJecoSvN931ahGX2x83HemXF2NBv3+kPO7VZ94Wisaf8cUrbF9JtMhvv5jmtl6TAN8VNVfQPG7Y16TqjDZUVjEv6xQYO8ZPr7loU5lUaE+7v/MKF/NGV5DYU2dToyk6ZDj82YpM/XJPVWkeb+6eC9X/3mpdoVQMB1wYrVvNYMEkcLCjgTmDDbk2Oct/dWsHap1SLSwc93iQ63QpYd7m0C1n5+kQZp7q+74EPZB6YJfTOIzs59fdxVew8F9rNj6/VoYnuuCm531f9yvNwi2g8JYXrxcVuHP5VO9c9XcD1TBXyZ9i/EmbbUDozLJ19ohnp13ZV9Oket6rYOZ2fc6XLxFkzmBv6rN5D6wAA7XN8Dmjn8nXHq8W5O+j6ie2z1mHP9sGzKt7w9nl6ETtvpefyUklh1MkqIIJMh1lGXVPFyds33v1lIztlTdjJx7XMcyvAje9hx6faZcOv44gwZljO0Y//6Vbv+zlHctjHeTsyE8pYqdozTrxpW8Y+dhSJHcwpm3/znRO9sovm8/N/RgGH8Atdyb/xaJ17byvcyb8Sn5C5pGRXSLyxVnbxwpgBmPZEL/DyCxiM3GipiqIoiqIoijIM9l3zGugftNxTc/rCO9nZsxkiZE5wDtshbvoo21umCtwbePF6vsykmOJ5oWUA9E2QoMSSWpFEgxoq6K8hAoCUhJaJ9/li/6R4/95u3pYWw7MZizNnImPB1DKeadvwVIALrv9yJ8sk1pKpzW5P8xTTalADWVKfZiMtU56YqHvsr5YHnH1KjZ2oXcof/rSqu1I8a80ne8aITMNti6zfXOmVzRugeS18brX3vSzA2pDSQH/bnVBuZVAcNezMPtTN99YLBu7r4qGm/gka7mxwdl9Fkn45HWEtRLowM/FFLhKcOqnf/37NmNUMdIjmFb6wQy78lfwv1Y34FJ8hCYRu903FfJoksXX1NLVW++BLxxzexGFurG17v+vs5f3ScqCZIfXIIc0rEm6cKI3wWFVbyJrAsFx3+WtOm+2JQuRset0zY+Nqrt+ZE1fx8R18fLLKF3qqdcDzKIcIFIk/hj+JRQ/3GZpQkbF/skhWQKSegYTYksed5tVqCr3zRVw7sdrYlNgeBuey5vK0QhdC8T+n8m94lq6B3NBntcznzuRP1ZySun74bFZvPvOj47yysld3AgDSu1gTaFd5v9PzSW+fmb9hLWzPSWw7HepxbWXF7EMAAJV3sqav4gnWNppat7LU9zMe+He+h++V3xfA9uVgLR+Xahi9xCEUjSJYNxvth7gE0DbEWqhL0grLvZ758FHePotEK9p7NtulxitcfZOx/mEQrc1065FuVbqqhvtpWwePfbTOhVF7aClrYStW8/6xDdyHg85EGS3Hy83r4B/xhwm0qbFtkiX7cLa+SwDQ90qk37NnILnRUhVFURRFURRlGOjLq6IoiqIoipI3jMxsgDikVT/D4cFMAUzm9tC0qQCAnvkutEnzQlZJ90wUZwXRGtulEsCFm0mWiKrZZ9SLiCylyHJ/2VReho+G3TU2t/FyTSoZ7Lcv/6Ccs0dME8QJoqnThUyqPu4wYEVmJqrRYFYx55tuSTvHraog//b2eDkAoHlB5hyj27DcSpEZjiI1hAlBQGKI2H3s58Kwb7lpgJVC0OdlYpf6eqpZXtFBf2n8eURCN00WZzSbKSy6M3MJ1mL6MkOXxai/CUqoK2OXAwq7xN0rhvYm5ExaIq392+LLa+u871Mm8XJ1byn3s1AXL/0NLu3cIFUli6YhGyrLDYt27Fm5g0Pw1LVs98qShbyMbfOfp70lMNf/vNz14kiSLkj7yliWAVlOozCXFRW5Pm3K2ITolfY6/t83dlmHLeus2TOZx43o63uq8f6jfJWTZeVh3HHKwzzW/b7tYABAeuPWzANTmc+P6Q+xfD52LnuJ3lLEjl7l1Z3ePp3NmcvvuQKFszxum3ls6jlsekZRqNu2R36WJUPcnwIV5e6cIibPjCtLqCzrkJhayyZST/c4B+DOWQOcgHwO1+OZRbHlcDFhanEys47an5vAz+JlSed4W3/2FADAhLd5aT0hYxBOaPX2SbzK4R1b5vOI1FfmRqZeMT+slXeUxlPZWTrY5+TRsJ2XzUOHcjtO7XCZOk0lL5GnJ0uoqVE0G0AqBbR1IBlzYaw6pvO1d00Wc0kZVkrecuPD6i/0N4dKR93YY9+zrIlYgVzujPvc8UXv8E4TNrgsZ4NhW8iUG7d424JVYp4nmd4QyfIksOalWcwQUw27EDCD2+ip5lVRFEVRFEXJG0ameTWZs7BQHc8Ye+axoXKimGc8fUXuvdiGkOmo40+/M1ZA1GGhrv5OIn2lbh/r5OA5Rvi1F2J0nejjA/skXE9rgwuOHS7lWZENp9XV6mZM4SLeVl3Os/e2bi5bWOXyZW+rmZthGD9aRCUhejpL2csbeIacnpkl5JUIympd/NrW4BBZzu1+MTmuWVRC88LOMrpwR//fi/pi/wTIal7F2WLQXxp/lnVy7uuPl/8TACA26kjOGTygdLbAys5ZhuWSzI3032NGQrQQZeu4TRSd5hxqArdW9du3dIXTvR91KM+639zKrSJvnPpE20RdPBwGfMr3+Fy+52Ytjyf+MFSBAcpB69yV9isYpCtax5NAjxsXrTbWalDtvlPKXLgkapfwRqtZa1RQ7PpmTytr1GzQ/r4S/syl1ZDa37zkfU9+rAwA0Cuq/TlRdrK570IXKqvk7hf5C2XqVYpWcND1Bzs5WLrVNvlDiyVLcz0wG/qFS0u3s1PMrqP4rvkdd0PdMrYXWYcXWWX0rQ4FUv3H+nSfWyUKpayA+vfDbX3OCWnWnJ39r80XKosiEkJuHDSvlS9xG6l52l1f65G8YvvljRcBAKJNTg5NB0k4sShfvx17yLeMmBanpFC3hHDyD2W22qIdt3046BsLjIwPkRJ5n1jl5Joo4c4c2MarCKPZCk0yiVTDLpT9yWlzy0bx/IOxr3c91bR7VK5jMFTzqiiKoiiKouQNexUqq/MiFx6nczLP9gPymm5nM542AQBJOKpAUjQcnW7GkiySWVCtzFVsUcTNQIOtYnsjr9rBYl9wYZl1JyTUVU8Xz2CD7U6TGK0efA6RaOUZ0y4JT2O1s+URp52rT5gMO9DRoiDIM+V4luQOkXWsBa48bmdGWdGABAR+bav9ns321ZaFRdfbZWw4IzdXjGxgrbO1GT0i6jfy5HMmipDz3LWKw4Z88SQOf9IscTfOnv+mt8/qzMMymBC0NnXcxoKZivADiguPfgUA8GJdHQDgvkW3eWWffugMAG6loGqlE8YhhRxe7uEvcPrGwCq215z+5Fhe7b4Tr+F+ZsewlMtPguJSHgfMm7zR2u0DQLJYpCCaTy9Qui9CmLUps4sX/tSxnubVBuxO8JeUL1RWai6vaEU38s4li9u9su4YLwE8sInD/ARLc0/T7dfadSdZMJMLWLPcLclDOj/mNM0ld8txiUzb8+Q2tjc+qXAdAOA/pnHbqyp0dnGt8QkZx+UMUdGJ+2+ThDjqLefPoE/jHG7lVaBkCR+XjkoD86WHtTaZ6VCWey/2r4ES7ocpSW7wRscUd0lBSSvqHeNroDR+7an5UK5XxWqnY2w6jK+naSfbflZNdR21R5JTdDawbBIlvO/RE5099crp3E/sCnByshu7wjE+vm8a20x3TuPjveQiAErW8b3pCHG/q9nqtNxtdaKqrZH21+hLZqKMCap5VRRFURRFUfKGEWle0xVF6DjjWCQ/6WwZOteyR1mswaaU5O1+ezdrQ2YD4fpnnuFOGyxXPCNl4pco8WkirSOl2L4a/+RQPHwnSEDdhZViFzLH7VMa5hlsyKo/prmynXH2NK6J8oU39/Gsqr7bzfgK6rsQ6MtmlbrvNIsKM24yZ7lWFpdMe9Xb1pnmuoRp+MG4wz51T1p+JyHzFmfP6TSv3QezZ/UzHWxbdnLsFa+sLc03M1U4RqroUaTkOdaoxU6WGbNEkf9ercv3+1Ecn3mg0Gt4Zh3zbH7F9mxsmsL4I5qWqKggz5vCQb1va3uPt0u6q3+oBRs4HQAWRNkm8fKDOIj4reHjkA/sOoLva6pAtFA+m/qDKlmDsqlFIn+c5DSvQatVTfe3WfWnlxWTdm+bv8yuTgV7pe+L3X5vyg3Lze9h7VK4g//vjDuLVirgdlleyNrhTUfwWOZScOQWkwp4jK6RyjQm+Xq/Mv8pb597MDHzwAFUS1rQsxdxsoLSkFslWxObPDoXOwZQTDSoQTfWWw2z9QPx27wGdrO8ElPEa9wOuWXOnyMZtW3QntCXMlW8uykgKdPFBnFbp/NUOLFmPQDgZUgKd196WQTHL+HDv5zCiRRuJ5eS9IkP/QQAcOlb/wcAULHMvYdQmh/q5c9tBgD0zWH72CcXLPT2mb+C5WkC3O562p3m1vbLwDO8SldVtRgAENvttKvdtfz8uPvKXwAAvjrnIq/svEpOWfwA3svHrxp+XZW9QzWviqIoiqIoSt6gL6+KoiiKoihK3jAis4FgRy/Kl27AmsWzvG01i3hZbcbR/fNpx5MuXkxDNy9bNLXwckey1Xk0hMWxKi2JB+zquZng1PWHz+IQPNUxXtqfJTl7ARcy6toqdr35990cUuaxBrdc8JN5DwIAJgRtOJLMJe9uWSJ+tJtDf62Lu0QKz5ZPgQmNzXt+T4rlFMviEWZlckTBRm9bvRjhDwycnw3rsJUtSHxC5JbtPJvP43sS38my/G6NM1uwC+iJ8twPSTNpKbeTxmtYjtY57fne4XmbbUiwbIIDHN/MgTrlk35RJsuwtWF2pGlOFg96iA1NAwBF0pbmx9h8IBzM/TYCACHr7yNBvGsnucDmUwv5e+crvOTYdJSXBR7hNm4XLnSa9GFfcxFLFW9Z0hd1zgsPaMPxxMWxNelz2OqaztdU9wDL9omr/+SVLX6Nly1bJbxfbOde+d/uNx5/9SAAwC9OvwMA8Fp3HQBgS8rvZLVnc6S/dHAIvIOL2EGwPOgctpYEjsl6TE4gZjnZnH9DtVyH2zt8ZhP2OWUjXomJjn9p3zoZeuYoWUKMId3fzmnLWvdsmzTVju1ibJIjZgNPfJ/NBeY/9pa37dSiqwAABRu5U5VXuX4qj1EYMWsKN7E8I7tc2wqs5/ZSFeBnfOd0F/Owt9R6TbLMizfwu0a60D09UxG+f2c/+hUAQPULrr89KOec+A92ENv/wcXefRyoj2FFURRF1WZDAwAADOBJREFUURTlAGREU3UbLHf21Zmpz9orOMRE+/t5Vtwyz83aQotZK3vIVA51Mn2+09JOifL3geGdEml3aW91ssH642sWAAAqnnKG1tV3rQQAnNHlEg9wxVyask8+8TEAwPuqOc3ZSl+okJ1drEnZ3cWzsGTSJj1wvz/v9fWg7rGJj2SdM6oCkYyy9FyePZb7wmLZpAJWy9Un849siQnsNn9ZeoAW0Wle3TymfBrPaBtXcUiS6GFu9pmWQP0I5b7XUuotvt9rE6xVqAzwrLw66JyOAodym0qvfCfj+A5xZiui/vNoM34Kif1Cm2QVmRPlkGnb0oOHHwr6HLbihttySYCdClt38UpLzZhc5egx6WecbtImUwxWuBSjqw7iFKaBFey81n7xYq8sIoqfpFXkS9eKuMhPiFdKKCPpQumo64u2HdkkLpDUrzubnXb3sCPYoSb+S9Z+n7P4HK+ssoU13AOd6HKVhT/nlZDWU3mstckVFhTs8PZ589BTAGTvj5aNvTwuzYzyql8s4AvM35rD2mca4ODn4+DJLIPHmxdl7J+K2RSa/JEu8qUltcO2Vfr7taWiRTRF/Z+NwU7faok8W2wqWPgTEmRL57mfaJslKaZPWuBtI3HUq1jDn1vPdI5n885eCwBYeTjLz0ziMWjZyT/x9jmh/GoAQOEU1qqeM/OfXlmVOBH+YRKHAkwcIs+KChearmspt9vCKn4ud9c6p27bv1NVsm3rtmHXVdk7VPOqKIqiKIqi5A2jNk21aROL7uPZTDarQqsfeNu37W2UZNlzIKzimIPXMkqGo/8LvJ/tUJ6GnYE2e2VR+T5UgJUUAGPGxn6vMylJFbIEhK6UlLW1QVfLVtHO9A2YdyR86kCrh7BabH+ygrRM1QMSW8VqZdcknPbm3xY8DAD45vpLM67JZiMMFuSHPSPgNK4x0aBOCDjtQvt8nikXr8w87qlOnsV/pJTb3co+ns0f6JrXxj7uk9VlrHV4NjHPV9pfCx3o9acO5jYVsYqlZO4FzR8O/VLALpOVoUrWPicqXH2jLTaVJFc41ig2qz5FV9+E/slXqM/Xb61Joyy62JBZfr2ctfNfccxsAEDZn17cixrlBqm1GwAA7/TwaGtX3fw2qw0nsNa7Okt/tHQkefWpsIC1huUBd3wqumeb2fHG2k76qYxyHZ7d7PxJZidZsxyMi62r1a6mXBu027Kdk5K830CJ2MQZADAxxMsEgTLW9puE02JTLIbxwtqRd9e4V5SKSg6NVf4MX3P5c24g3rWW5Tb37zxW957I9tXH93zV22fuEh6/mw7luj5Q68Ik2rB1s37HCTDih0na+0q3bjRlCa/QrP3DkQCASJGTbFJCR9pUvqoVHHtUxoqiKIqiKEreoC+viqIoiqIoSt6Qw9bt7w56JKRYQ8o5ZU0P8bbof/FSZcOv3RxjoiyxxQeuXftWjayZgM2mFfDHZiEbaivV7zyzQ26t8/I17wMA1D0oS0iXuMPjYnYQCudoMBC/+YU4LHzixc8AAP5xwi8BAH7J7Tye959zb+aptveW9/vfmlhEW3J/aXJfOKGMnR+sM1+YBjcRCba57EbWdMWaZwR68mxubB1qfE4vRhxYWs5g0wlKZnGM7JHlfqlu90y39Fqwub/TS7zWJ8sBmQPteXp73LC8fBdnDmo6kZeOy1ykLNfWs4T+yxmy9Me7nuPMa//2/v8HAGhNuZBFdLZkTfr14Ke02Q8jpSzLsN+hMpC7sjCFvAyfrTudX8khqx5ffpC3Lb6QHYs7J3N7CPfI+NPksqxZv+a0DcXmc7KyGbbMAMerYNx9nx8WeVuHrb49h2DcHySKua6dU1z7OX8qO/G91s7Oi7svOswrs4/D0gjb4HRN5s/iNf72xzHpwmJlUvimuxHWQSzVwM7ose1svtI10TmrhmbV8ZdOllXvVCerSJHNPMm/m2cjX16iMlYURVEURVHyBtW8jjOVMXYmihs3Q+xM89Q4LUb4L8dneGWXlfLM8E8dHP4pTINrQLOGzxKnmj6ZqnaneRZ/aMSFP9vexBrHOTs7M47vleMOn8Jhz1oy9hhn/EG6xcmu+kEJL3YSy7PDF6rmi6c/BgB4FC48kaUgyDPrgY5vwd7c1e6MBi+0s3PQ8bWcHKMn5Q/jNiBk3C6XXzwuocUmSKisWGOezY1taKFkZp9qnScrDu3u3tvA6DavfNdsbi8FW5ymyyZA6Jko5474cs+TaHtiA5IbJJzc4gkeogtrsoTDshrXHNbAZtNiT39YnEVP48+GhAs5dFQtO9duGuKc9Z3cVycEeXx63Tc+UkXfPl/zWGHCEv4p7u6TDVFVHeRQTZXLfetCoqINyXhjNfvxardKFu7ispBonAPlTpbJctZo29BcgSJ2o659xfXhp3vY0Sk5h53ogq+6EGUUzQzfuL+YsIo/q5e6kFN3zT0aADA3/joAIFHinpmBBNeRqlhT2rhYnEebXV/aegbX3zpXFW92su49kttScBGvsHTNYjl2zHC/UfEWa3wnLuNtzQt9yZY6+HvBqk38G8OuqbK35NnTRVEURVEURXk3o5rXcealV3imVzLNzcYbUzxvK1nJQeKXLHCBvJYMGdRr7/GfdyZWAACMBPDfmHAa2CqZrP5zxRwAwDy8NCbXs7f00/SkWXNReieHF3rjB6yxqfSF1kkMEffqgXWHAAC+cexzAIAGiRPWNcnN+coyD8t7nnzqcADAjy59HIBLJZyN1G4Xdm5tL6e2rC5kLWHJltxPZJENL2A7nLYwPo01eoGIs5MLrmcNGIkiq3id2Ko3u77cukA0QhLiDmGf1k3EE+rkski72KqHnbwLIqzNrYixbXGg0NmHprulHdvVhjEK57cvmFTmNUX//jIA4MnreHyZXejSfZ9QyvbWG046FwAQeDYzPGJrB8t9Yoi1lR1pX9D+1vHTFu4R0Yz7XRACs+sAAOWBpwEAlbe8kHHYSMaYfhq/7ZzEwv6c/Qw94dJ9TxOb162ncbua/pwziA2M4+hWuok7Vf05U71tM2S1DzKutx7samtTI6dLuC2UT+fwml2tmamHbWi7ZIHTqhbEuJ8ly/n4RBH3qZ4pPolS/5W33hpXVlTPz5HkjobhV1LZJ1TzqiiKoiiKouQNqnkdZ6pf4dncpIuKvW1tafHgTo+v5spEuHlM8GkzywI8Mw115makfpMc3Fv2b63vAQDcNOkVb9vUENtPPXz2VQCA6EMve2VBSQ5RFWRbqRJJQ9lbmXu2haNJUDSJtt596eENExPDHDx8msitZHN8qN1zFpMlfeeCq1YDANZe77zBF57KWsLZxaw5XFo/FwDQl3R9ozbGwmzYzVqsqjJnu9pRzPbmk8pZg3joBNYsbeqq9PbZ1MJez/GbeWUk1p0l7WQ69zSuHkPY4S7fwZEUrjniUW9bl6QY3nIG26nXPZt5XFkxt6uJQam3z14/XN2TeUCO0DGHV356y5zGL1nF4/65t30dAFCHTM3rWPKFez4PACjfJLazM539sAmOn24r9NybAID4F+d7266b9SAA4D/AK2Jzb3O2uztO4HZDSR57Kot4VSLZ7PpS50xuL5QQHwZfMpGKAm5TJHlebdKHUKtvJe9lThHdfQRHy1i4YLNXtnlLneyUn6tN+YhqXhVFURRFUZS8QV9eFUVRFEVRlLxBzQbGmZKtvPTx3Ua3HLm7j5eSTFt7xv4UZocEb3mcRmf+QQG3lOWFCnqdw6acu+pSr2xqMRvC176Uo8sjQyxTPnnnYgDAouMWeNvK72NZlzyUmTO+bAmXva/kQwCA5i52apj87IEdCGX2b7cAAE4+5gIAQMsTk7yyyXh+0OO+9twl/f6f+9zyMbi6/UCWZfh0By/tz77atRNrALBqIZsLhA5n55C+WtcnW0r40/pgdcKtVUokNuxO8k6vbGGHt/JHV3v7TGp5ey8rkftM/RF/fvBzX/G22SXduqVDhLz6Cy8FH9N4JQAg0OZCk015KkfHJQCRDm5X6bAba0MtvLw99akhzLDGMBTa7LvZ1Id6uTGaUG6Yg5kE3//SB5w53ecaPgsAmIt/AgDo+RVe2WQZlmzPbb2Dl/anPu9MSracXwMAKN7ObaR0vTPhad7Jfa/gRTbbqHyLTTzKb8t8Bte8yM/AXWlnYlGzRdprDoarO1BRzauiKIqiKIqSN5AZwUyBiBoBbN7jjgcmM4wx1SM5QOU1fHmprLRtjQCV18hQeY0MldfI0LF++GjbGhmDymtEL6+KoiiKoiiKMp6o2YCiKIqiKIqSN+jLq6IoiqIoipI36MuroiiKoiiKkjfoy6uiKIqiKIqSN+jLq6IoiqIoipI36MuroiiKoiiKkjfoy6uiKIqiKIqSN+jLq6IoiqIoipI36MuroiiKoiiKkjf8f65rabeRKpTXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x864 with 9 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X, y = iter(test_iter).next()\n",
    "\n",
    "true_labels = get_fashion_mnist_labels(y.numpy())\n",
    "pred_labels = get_fashion_mnist_labels(tf.argmax(net(X), axis=1).numpy())\n",
    "titles = [true + '\\n' + pred for true, pred in zip(true_labels, pred_labels)]\n",
    "\n",
    "show_fashion_mnist(X[0:9], titles[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_623cj7f",
    "id": "740E43F6246C47078E68E5EB4399345A",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## softmax的简洁实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:43:02.263905Z",
     "start_time": "2020-03-29T08:42:58.867885Z"
    },
    "graffitiCellId": "id_s0m2c8a",
    "id": "94B0411C93ED4E6C83B5545D9339E395",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_7ny2q9r",
    "id": "21E9BF3FC62E42FAA0D601526BC27572",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 初始化参数和获取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:34.566785Z",
     "start_time": "2020-04-06T07:13:34.233678Z"
    },
    "graffitiCellId": "id_3ra54mv",
    "id": "08E33ED9A5684635874D11BFC0B488C6",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:36.026973Z",
     "start_time": "2020-04-06T07:13:35.810435Z"
    }
   },
   "outputs": [],
   "source": [
    "# 归一化数据\n",
    "x_train = x_train / 255.0\n",
    "x_test = x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:43:14.604708Z",
     "start_time": "2020-03-29T08:43:14.596729Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 28, 28), (60000,))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 6000 * 10 * 28 * 28\n",
    "x_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_pvkd7mm",
    "id": "7BCB961EF5464E1980619F1853FE2F84",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义网络模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:37.497692Z",
     "start_time": "2020-04-06T07:13:37.220112Z"
    },
    "graffitiCellId": "id_1s76ktf",
    "id": "01609B4BD2B444588B2C33751B48B9B0",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 第一层是Flatten，将28 * 28的像素值，压缩成一行 (784, ) 第二层还是Dense，因为是多分类问题，激活函数使用softmax\n",
    "net = keras.Sequential([keras.layers.Flatten(input_shape=(28, 28)), \n",
    "                         keras.layers.Dense(10, activation=tf.nn.softmax)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:40.101685Z",
     "start_time": "2020-04-06T07:13:40.052785Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Flatten in module tensorflow.python.keras.layers.core:\n",
      "\n",
      "class Flatten(tensorflow.python.keras.engine.base_layer.Layer)\n",
      " |  Flattens the input. Does not affect the batch size.\n",
      " |  \n",
      " |  If inputs are shaped `(batch,)` without a channel dimension, then flattening\n",
      " |  adds an extra channel dimension and output shapes are `(batch, 1)`.\n",
      " |  \n",
      " |  Arguments:\n",
      " |    data_format: A string,\n",
      " |      one of `channels_last` (default) or `channels_first`.\n",
      " |      The ordering of the dimensions in the inputs.\n",
      " |      `channels_last` corresponds to inputs with shape\n",
      " |      `(batch, ..., channels)` while `channels_first` corresponds to\n",
      " |      inputs with shape `(batch, channels, ...)`.\n",
      " |      It defaults to the `image_data_format` value found in your\n",
      " |      Keras config file at `~/.keras/keras.json`.\n",
      " |      If you never set it, then it will be \"channels_last\".\n",
      " |  \n",
      " |  Example:\n",
      " |  \n",
      " |  ```python\n",
      " |  model = Sequential()\n",
      " |  model.add(Convolution2D(64, 3, 3,\n",
      " |                          border_mode='same',\n",
      " |                          input_shape=(3, 32, 32)))\n",
      " |  # now: model.output_shape == (None, 64, 32, 32)\n",
      " |  \n",
      " |  model.add(Flatten())\n",
      " |  # now: model.output_shape == (None, 65536)\n",
      " |  ```\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      Flatten\n",
      " |      tensorflow.python.keras.engine.base_layer.Layer\n",
      " |      tensorflow.python.module.module.Module\n",
      " |      tensorflow.python.training.tracking.tracking.AutoTrackable\n",
      " |      tensorflow.python.training.tracking.base.Trackable\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, data_format=None, **kwargs)\n",
      " |  \n",
      " |  call(self, inputs)\n",
      " |      This is where the layer's logic lives.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          inputs: Input tensor, or list/tuple of input tensors.\n",
      " |          **kwargs: Additional keyword arguments.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor or list/tuple of tensors.\n",
      " |  \n",
      " |  compute_output_shape(self, input_shape)\n",
      " |      Computes the output shape of the layer.\n",
      " |      \n",
      " |      If the layer has not been built, this method will call `build` on the\n",
      " |      layer. This assumes that the layer will later be used with inputs that\n",
      " |      match the input shape provided here.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          input_shape: Shape tuple (tuple of integers)\n",
      " |              or list of shape tuples (one per output tensor of the layer).\n",
      " |              Shape tuples can include None for free dimensions,\n",
      " |              instead of an integer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          An input shape tuple.\n",
      " |  \n",
      " |  get_config(self)\n",
      " |      Returns the config of the layer.\n",
      " |      \n",
      " |      A layer config is a Python dictionary (serializable)\n",
      " |      containing the configuration of a layer.\n",
      " |      The same layer can be reinstantiated later\n",
      " |      (without its trained weights) from this configuration.\n",
      " |      \n",
      " |      The config of a layer does not include connectivity\n",
      " |      information, nor the layer class name. These are handled\n",
      " |      by `Network` (one layer of abstraction above).\n",
      " |      \n",
      " |      Returns:\n",
      " |          Python dictionary.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  __call__(self, inputs, *args, **kwargs)\n",
      " |      Wraps `call`, applying pre- and post-processing steps.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: input tensor(s).\n",
      " |        *args: additional positional arguments to be passed to `self.call`.\n",
      " |        **kwargs: additional keyword arguments to be passed to `self.call`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor(s).\n",
      " |      \n",
      " |      Note:\n",
      " |        - The following optional keyword arguments are reserved for specific uses:\n",
      " |          * `training`: Boolean scalar tensor of Python boolean indicating\n",
      " |            whether the `call` is meant for training or inference.\n",
      " |          * `mask`: Boolean input mask.\n",
      " |        - If the layer's `call` method takes a `mask` argument (as some Keras\n",
      " |          layers do), its default value will be set to the mask generated\n",
      " |          for `inputs` by the previous layer (if `input` did come from\n",
      " |          a layer that generated a corresponding mask, i.e. if it came from\n",
      " |          a Keras layer with masking support.\n",
      " |      \n",
      " |      Raises:\n",
      " |        ValueError: if the layer's `call` method returns None (an invalid value).\n",
      " |  \n",
      " |  __delattr__(self, name)\n",
      " |      Implement delattr(self, name).\n",
      " |  \n",
      " |  __getstate__(self)\n",
      " |  \n",
      " |  __setattr__(self, name, value)\n",
      " |      Support self.foo = trackable syntax.\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  add_loss(self, losses, inputs=None)\n",
      " |      Add loss tensor(s), potentially dependent on layer inputs.\n",
      " |      \n",
      " |      Some losses (for instance, activity regularization losses) may be dependent\n",
      " |      on the inputs passed when calling a layer. Hence, when reusing the same\n",
      " |      layer on different inputs `a` and `b`, some entries in `layer.losses` may\n",
      " |      be dependent on `a` and some on `b`. This method automatically keeps track\n",
      " |      of dependencies.\n",
      " |      \n",
      " |      This method can be used inside a subclassed layer or model's `call`\n",
      " |      function, in which case `losses` should be a Tensor or list of Tensors.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      class MyLayer(tf.keras.layers.Layer):\n",
      " |        def call(inputs, self):\n",
      " |          self.add_loss(tf.abs(tf.reduce_mean(inputs)), inputs=True)\n",
      " |          return inputs\n",
      " |      ```\n",
      " |      \n",
      " |      This method can also be called directly on a Functional Model during\n",
      " |      construction. In this case, any loss Tensors passed to this Model must\n",
      " |      be symbolic and be able to be traced back to the model's `Input`s. These\n",
      " |      losses become part of the model's topology and are tracked in `get_config`.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      inputs = tf.keras.Input(shape=(10,))\n",
      " |      x = tf.keras.layers.Dense(10)(inputs)\n",
      " |      outputs = tf.keras.layers.Dense(1)(x)\n",
      " |      model = tf.keras.Model(inputs, outputs)\n",
      " |      # Actvity regularization.\n",
      " |      model.add_loss(tf.abs(tf.reduce_mean(x)))\n",
      " |      ```\n",
      " |      \n",
      " |      If this is not the case for your loss (if, for example, your loss references\n",
      " |      a `Variable` of one of the model's layers), you can wrap your loss in a\n",
      " |      zero-argument lambda. These losses are not tracked as part of the model's\n",
      " |      topology since they can't be serialized.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      inputs = tf.keras.Input(shape=(10,))\n",
      " |      x = tf.keras.layers.Dense(10)(inputs)\n",
      " |      outputs = tf.keras.layers.Dense(1)(x)\n",
      " |      model = tf.keras.Model(inputs, outputs)\n",
      " |      # Weight regularization.\n",
      " |      model.add_loss(lambda: tf.reduce_mean(x.kernel))\n",
      " |      ```\n",
      " |      \n",
      " |      The `get_losses_for` method allows to retrieve the losses relevant to a\n",
      " |      specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        losses: Loss tensor, or list/tuple of tensors. Rather than tensors, losses\n",
      " |          may also be zero-argument callables which create a loss tensor.\n",
      " |        inputs: Ignored when executing eagerly. If anything other than None is\n",
      " |          passed, it signals the losses are conditional on some of the layer's\n",
      " |          inputs, and thus they should only be run where these inputs are\n",
      " |          available. This is the case for activity regularization losses, for\n",
      " |          instance. If `None` is passed, the losses are assumed\n",
      " |          to be unconditional, and will apply across all dataflows of the layer\n",
      " |          (e.g. weight regularization losses).\n",
      " |  \n",
      " |  add_metric(self, value, aggregation=None, name=None)\n",
      " |      Adds metric tensor to the layer.\n",
      " |      \n",
      " |      Args:\n",
      " |        value: Metric tensor.\n",
      " |        aggregation: Sample-wise metric reduction function. If `aggregation=None`,\n",
      " |          it indicates that the metric tensor provided has been aggregated\n",
      " |          already. eg, `bin_acc = BinaryAccuracy(name='acc')` followed by\n",
      " |          `model.add_metric(bin_acc(y_true, y_pred))`. If aggregation='mean', the\n",
      " |          given metric tensor will be sample-wise reduced using `mean` function.\n",
      " |          eg, `model.add_metric(tf.reduce_sum(outputs), name='output_mean',\n",
      " |          aggregation='mean')`.\n",
      " |        name: String metric name.\n",
      " |      \n",
      " |      Raises:\n",
      " |        ValueError: If `aggregation` is anything other than None or `mean`.\n",
      " |  \n",
      " |  add_update(self, updates, inputs=None)\n",
      " |      Add update op(s), potentially dependent on layer inputs. (deprecated arguments)\n",
      " |      \n",
      " |      Warning: SOME ARGUMENTS ARE DEPRECATED: `(inputs)`. They will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      `inputs` is now automatically inferred\n",
      " |      \n",
      " |      Weight updates (for instance, the updates of the moving mean and variance\n",
      " |      in a BatchNormalization layer) may be dependent on the inputs passed\n",
      " |      when calling a layer. Hence, when reusing the same layer on\n",
      " |      different inputs `a` and `b`, some entries in `layer.updates` may be\n",
      " |      dependent on `a` and some on `b`. This method automatically keeps track\n",
      " |      of dependencies.\n",
      " |      \n",
      " |      The `get_updates_for` method allows to retrieve the updates relevant to a\n",
      " |      specific set of inputs.\n",
      " |      \n",
      " |      This call is ignored when eager execution is enabled (in that case, variable\n",
      " |      updates are run on the fly and thus do not need to be tracked for later\n",
      " |      execution).\n",
      " |      \n",
      " |      Arguments:\n",
      " |        updates: Update op, or list/tuple of update ops, or zero-arg callable\n",
      " |          that returns an update op. A zero-arg callable should be passed in\n",
      " |          order to disable running the updates by setting `trainable=False`\n",
      " |          on this Layer, when executing in Eager mode.\n",
      " |        inputs: Deprecated, will be automatically inferred.\n",
      " |  \n",
      " |  add_variable(self, *args, **kwargs)\n",
      " |      Deprecated, do NOT use! Alias for `add_weight`. (deprecated)\n",
      " |      \n",
      " |      Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      Please use `layer.add_weight` method instead.\n",
      " |  \n",
      " |  add_weight(self, name=None, shape=None, dtype=None, initializer=None, regularizer=None, trainable=None, constraint=None, partitioner=None, use_resource=None, synchronization=<VariableSynchronization.AUTO: 0>, aggregation=<VariableAggregation.NONE: 0>, **kwargs)\n",
      " |      Adds a new variable to the layer.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        name: Variable name.\n",
      " |        shape: Variable shape. Defaults to scalar if unspecified.\n",
      " |        dtype: The type of the variable. Defaults to `self.dtype` or `float32`.\n",
      " |        initializer: Initializer instance (callable).\n",
      " |        regularizer: Regularizer instance (callable).\n",
      " |        trainable: Boolean, whether the variable should be part of the layer's\n",
      " |          \"trainable_variables\" (e.g. variables, biases)\n",
      " |          or \"non_trainable_variables\" (e.g. BatchNorm mean and variance).\n",
      " |          Note that `trainable` cannot be `True` if `synchronization`\n",
      " |          is set to `ON_READ`.\n",
      " |        constraint: Constraint instance (callable).\n",
      " |        partitioner: Partitioner to be passed to the `Trackable` API.\n",
      " |        use_resource: Whether to use `ResourceVariable`.\n",
      " |        synchronization: Indicates when a distributed a variable will be\n",
      " |          aggregated. Accepted values are constants defined in the class\n",
      " |          `tf.VariableSynchronization`. By default the synchronization is set to\n",
      " |          `AUTO` and the current `DistributionStrategy` chooses\n",
      " |          when to synchronize. If `synchronization` is set to `ON_READ`,\n",
      " |          `trainable` must not be set to `True`.\n",
      " |        aggregation: Indicates how a distributed variable will be aggregated.\n",
      " |          Accepted values are constants defined in the class\n",
      " |          `tf.VariableAggregation`.\n",
      " |        **kwargs: Additional keyword arguments. Accepted values are `getter`,\n",
      " |          `collections`, `experimental_autocast` and `caching_device`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        The created variable. Usually either a `Variable` or `ResourceVariable`\n",
      " |        instance. If `partitioner` is not `None`, a `PartitionedVariable`\n",
      " |        instance is returned.\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called with partitioned variable regularization and\n",
      " |          eager execution is enabled.\n",
      " |        ValueError: When giving unsupported dtype and no initializer or when\n",
      " |          trainable has been set to True with synchronization set as `ON_READ`.\n",
      " |  \n",
      " |  apply(self, inputs, *args, **kwargs)\n",
      " |      Deprecated, do NOT use! (deprecated)\n",
      " |      \n",
      " |      Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      Please use `layer.__call__` method instead.\n",
      " |      \n",
      " |      This is an alias of `self.__call__`.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor(s).\n",
      " |        *args: additional positional arguments to be passed to `self.call`.\n",
      " |        **kwargs: additional keyword arguments to be passed to `self.call`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor(s).\n",
      " |  \n",
      " |  build(self, input_shape)\n",
      " |      Creates the variables of the layer (optional, for subclass implementers).\n",
      " |      \n",
      " |      This is a method that implementers of subclasses of `Layer` or `Model`\n",
      " |      can override if they need a state-creation step in-between\n",
      " |      layer instantiation and layer call.\n",
      " |      \n",
      " |      This is typically used to create the weights of `Layer` subclasses.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        input_shape: Instance of `TensorShape`, or list of instances of\n",
      " |          `TensorShape` if the layer expects a list of inputs\n",
      " |          (one instance per input).\n",
      " |  \n",
      " |  compute_mask(self, inputs, mask=None)\n",
      " |      Computes an output mask tensor.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          inputs: Tensor or list of tensors.\n",
      " |          mask: Tensor or list of tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |          None or a tensor (or list of tensors,\n",
      " |              one per output tensor of the layer).\n",
      " |  \n",
      " |  compute_output_signature(self, input_signature)\n",
      " |      Compute the output tensor signature of the layer based on the inputs.\n",
      " |      \n",
      " |      Unlike a TensorShape object, a TensorSpec object contains both shape\n",
      " |      and dtype information for a tensor. This method allows layers to provide\n",
      " |      output dtype information if it is different from the input dtype.\n",
      " |      For any layer that doesn't implement this function,\n",
      " |      the framework will fall back to use `compute_output_shape`, and will\n",
      " |      assume that the output dtype matches the input dtype.\n",
      " |      \n",
      " |      Args:\n",
      " |        input_signature: Single TensorSpec or nested structure of TensorSpec\n",
      " |          objects, describing a candidate input for the layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Single TensorSpec or nested structure of TensorSpec objects, describing\n",
      " |          how the layer would transform the provided input.\n",
      " |      \n",
      " |      Raises:\n",
      " |        TypeError: If input_signature contains a non-TensorSpec object.\n",
      " |  \n",
      " |  count_params(self)\n",
      " |      Count the total number of scalars composing the weights.\n",
      " |      \n",
      " |      Returns:\n",
      " |          An integer count.\n",
      " |      \n",
      " |      Raises:\n",
      " |          ValueError: if the layer isn't yet built\n",
      " |            (in which case its weights aren't yet defined).\n",
      " |  \n",
      " |  get_input_at(self, node_index)\n",
      " |      Retrieves the input tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor (or list of tensors if the layer has multiple inputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_input_mask_at(self, node_index)\n",
      " |      Retrieves the input mask tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A mask tensor\n",
      " |          (or list of tensors if the layer has multiple inputs).\n",
      " |  \n",
      " |  get_input_shape_at(self, node_index)\n",
      " |      Retrieves the input shape(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A shape tuple\n",
      " |          (or list of shape tuples if the layer has multiple inputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_losses_for(self, inputs)\n",
      " |      Retrieves losses relevant to a specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor or list/tuple of input tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |        List of loss tensors of the layer that depend on `inputs`.\n",
      " |  \n",
      " |  get_output_at(self, node_index)\n",
      " |      Retrieves the output tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor (or list of tensors if the layer has multiple outputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_output_mask_at(self, node_index)\n",
      " |      Retrieves the output mask tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A mask tensor\n",
      " |          (or list of tensors if the layer has multiple outputs).\n",
      " |  \n",
      " |  get_output_shape_at(self, node_index)\n",
      " |      Retrieves the output shape(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A shape tuple\n",
      " |          (or list of shape tuples if the layer has multiple outputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_updates_for(self, inputs)\n",
      " |      Retrieves updates relevant to a specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor or list/tuple of input tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |        List of update ops of the layer that depend on `inputs`.\n",
      " |  \n",
      " |  get_weights(self)\n",
      " |      Returns the current weights of the layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Weights values as a list of numpy arrays.\n",
      " |  \n",
      " |  set_weights(self, weights)\n",
      " |      Sets the weights of the layer, from Numpy arrays.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          weights: a list of Numpy arrays. The number\n",
      " |              of arrays and their shape must match\n",
      " |              number of the dimensions of the weights\n",
      " |              of the layer (i.e. it should match the\n",
      " |              output of `get_weights`).\n",
      " |      \n",
      " |      Raises:\n",
      " |          ValueError: If the provided weights list does not match the\n",
      " |              layer's specifications.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  from_config(config) from builtins.type\n",
      " |      Creates a layer from its config.\n",
      " |      \n",
      " |      This method is the reverse of `get_config`,\n",
      " |      capable of instantiating the same layer from the config\n",
      " |      dictionary. It does not handle layer connectivity\n",
      " |      (handled by Network), nor weights (handled by `set_weights`).\n",
      " |      \n",
      " |      Arguments:\n",
      " |          config: A Python dictionary, typically the\n",
      " |              output of get_config.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A layer instance.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  activity_regularizer\n",
      " |      Optional regularizer function for the output of this layer.\n",
      " |  \n",
      " |  dtype\n",
      " |  \n",
      " |  dynamic\n",
      " |  \n",
      " |  inbound_nodes\n",
      " |      Deprecated, do NOT use! Only for compatibility with external Keras.\n",
      " |  \n",
      " |  input\n",
      " |      Retrieves the input tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one input,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input tensor or list of input tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |        AttributeError: If no inbound nodes are found.\n",
      " |  \n",
      " |  input_mask\n",
      " |      Retrieves the input mask tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one inbound node,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input mask tensor (potentially None) or list of input\n",
      " |          mask tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer is connected to\n",
      " |          more than one incoming layers.\n",
      " |  \n",
      " |  input_shape\n",
      " |      Retrieves the input shape(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one input,\n",
      " |      i.e. if it is connected to one incoming layer, or if all inputs\n",
      " |      have the same shape.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input shape, as an integer shape tuple\n",
      " |          (or list of shape tuples, one tuple per input tensor).\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer has no defined input_shape.\n",
      " |          RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  input_spec\n",
      " |  \n",
      " |  losses\n",
      " |      Losses which are associated with this `Layer`.\n",
      " |      \n",
      " |      Variable regularization tensors are created when this property is accessed,\n",
      " |      so it is eager safe: accessing `losses` under a `tf.GradientTape` will\n",
      " |      propagate gradients back to the corresponding variables.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of tensors.\n",
      " |  \n",
      " |  metrics\n",
      " |  \n",
      " |  name\n",
      " |      Returns the name of this module as passed or determined in the ctor.\n",
      " |      \n",
      " |      NOTE: This is not the same as the `self.name_scope.name` which includes\n",
      " |      parent module names.\n",
      " |  \n",
      " |  non_trainable_variables\n",
      " |  \n",
      " |  non_trainable_weights\n",
      " |  \n",
      " |  outbound_nodes\n",
      " |      Deprecated, do NOT use! Only for compatibility with external Keras.\n",
      " |  \n",
      " |  output\n",
      " |      Retrieves the output tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one output,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor or list of output tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |        AttributeError: if the layer is connected to more than one incoming\n",
      " |          layers.\n",
      " |        RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  output_mask\n",
      " |      Retrieves the output mask tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one inbound node,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Output mask tensor (potentially None) or list of output\n",
      " |          mask tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer is connected to\n",
      " |          more than one incoming layers.\n",
      " |  \n",
      " |  output_shape\n",
      " |      Retrieves the output shape(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has one output,\n",
      " |      or if all outputs have the same shape.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Output shape, as an integer shape tuple\n",
      " |          (or list of shape tuples, one tuple per output tensor).\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer has no defined output shape.\n",
      " |          RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  stateful\n",
      " |  \n",
      " |  trainable\n",
      " |  \n",
      " |  trainable_variables\n",
      " |      Sequence of trainable variables owned by this module and its submodules.\n",
      " |      \n",
      " |      Note: this method uses reflection to find variables on the current instance\n",
      " |      and submodules. For performance reasons you may wish to cache the result\n",
      " |      of calling this method if you don't expect the return value to change.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A sequence of variables for the current module (sorted by attribute\n",
      " |        name) followed by variables from all submodules recursively (breadth\n",
      " |        first).\n",
      " |  \n",
      " |  trainable_weights\n",
      " |  \n",
      " |  updates\n",
      " |  \n",
      " |  variables\n",
      " |      Returns the list of all layer variables/weights.\n",
      " |      \n",
      " |      Alias of `self.weights`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of variables.\n",
      " |  \n",
      " |  weights\n",
      " |      Returns the list of all layer variables/weights.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of variables.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from tensorflow.python.module.module.Module:\n",
      " |  \n",
      " |  with_name_scope(method) from builtins.type\n",
      " |      Decorator to automatically enter the module name scope.\n",
      " |      \n",
      " |      ```\n",
      " |      class MyModule(tf.Module):\n",
      " |        @tf.Module.with_name_scope\n",
      " |        def __call__(self, x):\n",
      " |          if not hasattr(self, 'w'):\n",
      " |            self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))\n",
      " |          return tf.matmul(x, self.w)\n",
      " |      ```\n",
      " |      \n",
      " |      Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose\n",
      " |      names included the module name:\n",
      " |      \n",
      " |      ```\n",
      " |      mod = MyModule()\n",
      " |      mod(tf.ones([8, 32]))\n",
      " |      # ==> <tf.Tensor: ...>\n",
      " |      mod.w\n",
      " |      # ==> <tf.Variable ...'my_module/w:0'>\n",
      " |      ```\n",
      " |      \n",
      " |      Args:\n",
      " |        method: The method to wrap.\n",
      " |      \n",
      " |      Returns:\n",
      " |        The original method wrapped such that it enters the module's name scope.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.module.module.Module:\n",
      " |  \n",
      " |  name_scope\n",
      " |      Returns a `tf.name_scope` instance for this class.\n",
      " |  \n",
      " |  submodules\n",
      " |      Sequence of all sub-modules.\n",
      " |      \n",
      " |      Submodules are modules which are properties of this module, or found as\n",
      " |      properties of modules which are properties of this module (and so on).\n",
      " |      \n",
      " |      ```\n",
      " |      a = tf.Module()\n",
      " |      b = tf.Module()\n",
      " |      c = tf.Module()\n",
      " |      a.b = b\n",
      " |      b.c = c\n",
      " |      assert list(a.submodules) == [b, c]\n",
      " |      assert list(b.submodules) == [c]\n",
      " |      assert list(c.submodules) == []\n",
      " |      ```\n",
      " |      \n",
      " |      Returns:\n",
      " |        A sequence of all submodules.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.training.tracking.base.Trackable:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(keras.layers.Flatten)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_hn4au8n",
    "id": "EC61B11479EA443E812CCF03CC85D871",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:42.259965Z",
     "start_time": "2020-04-06T07:13:42.256974Z"
    },
    "graffitiCellId": "id_knh2n50",
    "id": "EFA9FA88D866458F913194664286D8AB",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "loss = 'sparse_categorical_crossentropy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:43.042567Z",
     "start_time": "2020-04-06T07:13:43.039575Z"
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow import losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:43.685668Z",
     "start_time": "2020-04-06T07:13:43.682169Z"
    }
   },
   "outputs": [],
   "source": [
    "# SparseCategoricalCrossentropy 已经除以了 batch_size,所以sgd 不需要再除以 batch_size 了\n",
    "# 默认，SUM_OVER_BATCH_SIZE，Scalar SUM divided by number of elements in losses. \n",
    "# https://stackoverflow.com/questions/58159154/how-to-calculate-categorical-cross-entropy-by-hand\n",
    "cross_entropy = losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T09:58:19.490656Z",
     "start_time": "2020-03-29T09:58:19.485670Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class SparseCategoricalCrossentropy in module tensorflow.python.keras.losses:\n",
      "\n",
      "class SparseCategoricalCrossentropy(LossFunctionWrapper)\n",
      " |  Computes the crossentropy loss between the labels and predictions.\n",
      " |  \n",
      " |  Use this crossentropy loss function when there are two or more label classes.\n",
      " |  We expect labels to be provided as integers. If you want to provide labels\n",
      " |  using `one-hot` representation, please use `CategoricalCrossentropy` loss.\n",
      " |  There should be `# classes` floating point values per feature for `y_pred`\n",
      " |  and a single floating point value per feature for `y_true`.\n",
      " |  \n",
      " |  In the snippet below, there is a single floating point value per example for\n",
      " |  `y_true` and `# classes` floating pointing values per example for `y_pred`.\n",
      " |  The shape of `y_true` is `[batch_size]` and the shape of `y_pred` is\n",
      " |  `[batch_size, num_classes]`.\n",
      " |  \n",
      " |  Usage:\n",
      " |  \n",
      " |  ```python\n",
      " |  cce = tf.keras.losses.SparseCategoricalCrossentropy()\n",
      " |  loss = cce(\n",
      " |    tf.convert_to_tensor([0, 1, 2]),\n",
      " |    tf.convert_to_tensor([[.9, .05, .05], [.5, .89, .6], [.05, .01, .94]]))\n",
      " |  print('Loss: ', loss.numpy())  # Loss: 0.3239\n",
      " |  ```\n",
      " |  \n",
      " |  Usage with the `compile` API:\n",
      " |  \n",
      " |  ```python\n",
      " |  model = tf.keras.Model(inputs, outputs)\n",
      " |  model.compile('sgd', loss=tf.keras.losses.SparseCategoricalCrossentropy())\n",
      " |  ```\n",
      " |  \n",
      " |  Args:\n",
      " |    from_logits: Whether `y_pred` is expected to be a logits tensor. By default,\n",
      " |      we assume that `y_pred` encodes a probability distribution.\n",
      " |      Note: Using from_logits=True may be more numerically stable.\n",
      " |    reduction: (Optional) Type of `tf.keras.losses.Reduction` to apply to loss.\n",
      " |      Default value is `AUTO`. `AUTO` indicates that the reduction option will\n",
      " |      be determined by the usage context. For almost all cases this defaults to\n",
      " |      `SUM_OVER_BATCH_SIZE`.\n",
      " |      When used with `tf.distribute.Strategy`, outside of built-in training\n",
      " |      loops such as `tf.keras` `compile` and `fit`, using `AUTO` or\n",
      " |      `SUM_OVER_BATCH_SIZE` will raise an error. Please see\n",
      " |      https://www.tensorflow.org/tutorials/distribute/custom_training\n",
      " |      for more details on this.\n",
      " |    name: Optional name for the op.\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      SparseCategoricalCrossentropy\n",
      " |      LossFunctionWrapper\n",
      " |      Loss\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, from_logits=False, reduction='auto', name='sparse_categorical_crossentropy')\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from LossFunctionWrapper:\n",
      " |  \n",
      " |  call(self, y_true, y_pred)\n",
      " |      Invokes the `LossFunctionWrapper` instance.\n",
      " |      \n",
      " |      Args:\n",
      " |        y_true: Ground truth values.\n",
      " |        y_pred: The predicted values.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Loss values per sample.\n",
      " |  \n",
      " |  get_config(self)\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from Loss:\n",
      " |  \n",
      " |  __call__(self, y_true, y_pred, sample_weight=None)\n",
      " |      Invokes the `Loss` instance.\n",
      " |      \n",
      " |      Args:\n",
      " |        y_true: Ground truth values. shape = `[batch_size, d0, .. dN]`\n",
      " |        y_pred: The predicted values. shape = `[batch_size, d0, .. dN]`\n",
      " |        sample_weight: Optional `sample_weight` acts as a\n",
      " |          coefficient for the loss. If a scalar is provided, then the loss is\n",
      " |          simply scaled by the given value. If `sample_weight` is a tensor of size\n",
      " |          `[batch_size]`, then the total loss for each sample of the batch is\n",
      " |          rescaled by the corresponding element in the `sample_weight` vector. If\n",
      " |          the shape of `sample_weight` is `[batch_size, d0, .. dN-1]` (or can be\n",
      " |          broadcasted to this shape), then each loss element of `y_pred` is scaled\n",
      " |          by the corresponding value of `sample_weight`. (Note on`dN-1`: all loss\n",
      " |          functions reduce by 1 dimension, usually axis=-1.)\n",
      " |      \n",
      " |      Returns:\n",
      " |        Weighted loss float `Tensor`. If `reduction` is `NONE`, this has\n",
      " |          shape `[batch_size, d0, .. dN-1]`; otherwise, it is scalar. (Note `dN-1`\n",
      " |          because all loss functions reduce by 1 dimension, usually axis=-1.)\n",
      " |      \n",
      " |      Raises:\n",
      " |        ValueError: If the shape of `sample_weight` is invalid.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from Loss:\n",
      " |  \n",
      " |  from_config(config) from builtins.type\n",
      " |      Instantiates a `Loss` from its config (output of `get_config()`).\n",
      " |      \n",
      " |      Args:\n",
      " |          config: Output of `get_config()`.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A `Loss` instance.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from Loss:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(losses.SparseCategoricalCrossentropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_bdskdx7",
    "id": "FFD0D8B3BB1B4F6582610743373C576E",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 定义优化函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:13:47.125475Z",
     "start_time": "2020-04-06T07:13:47.122453Z"
    },
    "graffitiCellId": "id_6l357pq",
    "id": "7DF366DFD7D349B59B0CA9D3211EB140",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.SGD(0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "graffitiCellId": "id_kzma3d6",
    "id": "6A660AA6E71541EA85B4955AC0DCC66C",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方式一，tf compile api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:45:57.062228Z",
     "start_time": "2020-03-29T08:45:54.287587Z"
    },
    "graffitiCellId": "id_0puuqtc",
    "id": "A49C9427C0E14DB9914E8970D98C422B",
    "jupyter": {},
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 1s 19us/sample - loss: 0.7979 - accuracy: 0.7376\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 0s 5us/sample - loss: 0.5765 - accuracy: 0.8090\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 0s 6us/sample - loss: 0.5294 - accuracy: 0.8236\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 0s 6us/sample - loss: 0.5041 - accuracy: 0.8312\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 0s 5us/sample - loss: 0.4881 - accuracy: 0.8352\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1870d8b1518>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])\n",
    "net.fit(x_train, y_train, epochs=5, batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T08:46:14.839955Z",
     "start_time": "2020-03-29T08:46:14.621512Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 20us/sample - loss: 0.5268 - accuracy: 0.8147\n",
      "Test Acc: 0.8147\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = net.evaluate(x_test, y_test)\n",
    "print('Test Acc:',test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方式二"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-04-06T07:14:15.628081Z",
     "start_time": "2020-04-06T07:14:13.444085Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (256, 10) <dtype: 'uint8'> (256,)\n",
      "<dtype: 'float32'> (96, 10) <dtype: 'uint8'> (96,)\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'evaluate_accuracy' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-13-18427dc1ab67>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     21\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     22\u001b[0m \u001b[0mtrainer\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkeras\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0moptimizers\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSGD\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlr\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 23\u001b[1;33m \u001b[0mtrain_ch3\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnet\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrain_iter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtest_iter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcross_entropy\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnum_epochs\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mlr\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrainer\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtrainer\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-13-18427dc1ab67>\u001b[0m in \u001b[0;36mtrain_ch3\u001b[1;34m(net, train_iter, test_iter, loss, num_epochs, batch_size, params, lr, trainer)\u001b[0m\n\u001b[0;32m     17\u001b[0m             \u001b[0mtrain_acc_sum\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreduce_sum\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0margmax\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_hat\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcast\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mint64\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mint64\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     18\u001b[0m             \u001b[0mn\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 19\u001b[1;33m         \u001b[0mtest_acc\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mevaluate_accuracy\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtest_iter\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnet\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     20\u001b[0m         \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'epoch %d, loss %.4f, train acc %.3f, test acc %.3f'\u001b[0m\u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrain_l_sum\u001b[0m \u001b[1;33m/\u001b[0m \u001b[0mn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtrain_acc_sum\u001b[0m \u001b[1;33m/\u001b[0m \u001b[0mn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtest_acc\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     21\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'evaluate_accuracy' is not defined"
     ]
    }
   ],
   "source": [
    "# 迭代周期，学习率\n",
    "num_epochs, lr = 5, 0.1\n",
    "def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params=None, lr=None, trainer=None):\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0\n",
    "        for X, y in train_iter:\n",
    "            with tf.GradientTape() as tape:\n",
    "                y_hat = net(X, training=True)\n",
    "                print(y_hat.dtype, y_hat.shape, y.dtype,y.shape)\n",
    "                l = loss(y, y_hat)\n",
    "                #print(params)\n",
    "                grads = tape.gradient(l, net.trainable_variables)\n",
    "                # tf.keras.optimizers.SGD 直接使用是随机梯度下降 theta(t+1) = theta(t) - learning_rate * gradient\n",
    "                trainer.apply_gradients(zip(grads, net.trainable_variables))  \n",
    "                \n",
    "            train_l_sum += l.numpy()\n",
    "            train_acc_sum += tf.reduce_sum(tf.cast(tf.argmax(y_hat, axis=1) == tf.cast(y, dtype=tf.int64), dtype=tf.int64)).numpy()\n",
    "            n += y.shape[0]\n",
    "        test_acc = evaluate_accuracy(test_iter, net)\n",
    "        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'% (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))\n",
    "\n",
    "trainer = tf.keras.optimizers.SGD(lr)\n",
    "train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, batch_size, lr=lr, trainer=trainer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "191.594px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
